# US 10,482,583 B1 · Generating and displaying blur in images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,583 B1 - Document ID: US10482583B1 - Status: Granted - Kind: B1 - Application number: 15/976,818 - Filed: 2018-05-10 - Published: 2019-11-14 - Granted: 2019-11-19 - Earliest claimed: 2018-05-10 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Austin Suszek - Web page: https://patentgrep.com/patent/US10482583B1 ## CPC - G06T 5/70: Denoising; Smoothing - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 5/20: using local operators - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/194: involving foreground-background segmentation - G06T 7/50: Depth or shape recovery - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/20092: Interactive image processing based on input by user - G06T 2207/20212: Image combination - G06T 2210/62: Semi-transparency - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Implementations relate to generating and displaying blur in images. In some implementations, a method includes generating a plurality of mipmap images based on an input image, including applying a blur to a respective plurality of pixels derived from the input image for each mipmap image. In some examples, the blur is at least partially based on depth data for the image. Parameter data is obtained that indicates an output focal plane depth for an output focal plane of an output image and an output focal range in front of the output focal plane. Output pixel values of the output image are generated, including determining blurred pixel values based on one or more of the mipmap images selected based on the output focal plane depth and the output focal range. The blurred pixel values are based on particular pixels associated with a depth outside the output focal range. ## Claims ### Claim 1 A computer-implemented method to provide blurring in an image, the method comprising: obtaining an input image; generating a plurality of mipmap images based on the input image, wherein generating each mipmap image of the plurality of mipmap images includes applying a blur to a respective plurality of pixels derived from the input image; obtaining parameter data that indicates an output focal plane depth for an output focal plane of an output image and that indicates an output focal range in front of the output focal plane; and generating output pixel values of the output image that include output blur, wherein generating the output pixel values includes determining blurred pixel values based on one or more selected mipmap images selected from the plurality of mipmap images based on the output focal plane depth and the output focal range, and wherein the blurred pixel values are based on particular pixels associated with a depth outside the output focal range. ### Claim 2 (depends on 1) The computer-implemented method of claim 1 wherein generating the plurality of mipmap images includes: generating a plurality of background mipmap images based on the input image, wherein generating each of the background mipmap images includes applying a respective background blur to each of a plurality of background pixels derived from the input image, wherein each background pixel has a respective depth behind a respective focal plane associated with the background pixel; and generating a plurality of foreground mipmap images based on the input image, wherein generating each of the foreground mipmap images includes applying a respective foreground blur to each of a plurality of foreground pixels derived from the input image, wherein each foreground pixel has a respective depth in front of a respective focal plane associated with the foreground pixel, wherein the one or more selected mipmap images include one or more of the background mipmap images and one or more of the foreground mipmap images. ### Claim 3 (depends on 2) The computer-implemented method of claim 2 wherein generating output pixel values includes: determining opaque pixel values based on at least one background mipmap image selected based on the output focal plane depth; determining overlay pixel values based on at least one foreground mipmap image selected based on the output focal range; and combining the opaque pixel values with corresponding values of the overlay pixel values to determine the output pixel values of the output image. ### Claim 4 (depends on 3) The computer-implemented method of claim 3 wherein determining the opaque pixel values based on the at least one background mipmap image includes, for each background pixel derived from the input image: selecting one or more background mipmap images from the plurality of background mipmap images based on the depth of the background pixel and based on the output focal plane depth; and using the one or more background mipmap images to determine a particular opaque pixel value corresponding to the background pixel. ### Claim 5 (depends on 3) The computer-implemented method of claim 3 wherein applying the respective foreground blur to each of the plurality of foreground pixels includes: applying the respective foreground blur to each foreground pixel based on surrounding pixels of the input image that surround the foreground pixel, and based on a foreground focal plane depth of a foreground focal plane associated with the foreground mipmap image. ### Claim 6 (depends on 5) The computer-implemented method of claim 5 wherein determining the overlay pixel values based on at least one foreground mipmap image includes, for each foreground pixel derived from the input image: selecting one or more foreground mipmap images of the plurality of foreground mipmap images based on the output focal plane depth and based on the output focal range; and using the one or more foreground mipmap images to determine a particular overlay pixel value corresponding to the foreground pixel. ### Claim 7 (depends on 3) The computer-implemented method of claim 3 wherein the combining includes using transparency values associated with the overlay pixel values, the transparency values indicating a respective transparency of the overlay pixel values. ### Claim 8 (depends on 1) The computer-implemented method of claim 1 wherein generating the plurality of mipmap images includes downscaling the input image to a plurality of downscaled images, and wherein applying the blur to the respective plurality of pixels includes applying the blur to a respective plurality of downscaled pixels of the downscaled images. ### Claim 9 (depends on 2) The computer-implemented method of claim 2 wherein generating the background mipmap images includes downscaling the input image to a plurality of background downscaled images and applying the respective background blur to each of the background pixels of the background downscaled images, wherein generating the foreground mipmap images includes downscaling the input image to a plurality of foreground downscaled images and applying the respective foreground blur to each of the foreground pixels of the foreground downscaled images, and wherein at least two of the background downscaled images are downscaled by different scale levels, and wherein at least two of the foreground downscaled images are downscaled by different scale levels. ### Claim 10 (depends on 2) The computer-implemented method of claim 2 wherein applying the respective background blur to each of the plurality of background pixels includes: applying the respective background blur to each background pixel based on surrounding pixels derived from the input image that surround the background pixel, based on a depth of the background pixel into a scene depicted by the input image, and based on a focal plane associated with the background pixel, wherein the focal plane is determined based on the depth of the background pixel and a background mipmap blur level assigned to the background mipmap image. ### Claim 11 (depends on 1) The computer-implemented method of claim 1 wherein at least one of the output focal plane depth or the output focal range are based on user input. ### Claim 12 A computer-implemented method to provide blurring in an image, the method comprising: obtaining an input image; generating a plurality of mipmap images based on the input image, wherein generating the plurality of mipmap images includes, for each mipmap image: applying a blur to each particular pixel of a plurality of pixels derived from the input image for the mipmap image, wherein the blur is applied to each particular pixel based on a depth of the particular pixel into a scene depicted by the input image and based on a respective focal plane associated with the particular pixel; obtaining data indicating an output focal plane depth for an output focal plane of an output image; determining blurred output pixel values of the output image using at least one selected mipmap image of the plurality of mipmap images, including, for each designated pixel of a set of pixels of the input image, selecting the at least one selected mipmap image based on the depth of the designated pixel and based on the output focal plane depth; and causing the output image to be displayed on a display device. ### Claim 13 (depends on 12) The computer-implemented method of claim 12 wherein generating the plurality of mipmap images includes, for each mipmap image, downscaling the input image to a downscaled image, wherein applying the blur to each particular pixel of the plurality of pixels of the mipmap image includes applying the blur to each particular pixel of a plurality of pixels of the downscaled image. ### Claim 14 (depends on 12) The computer-implemented method of claim 12 wherein generating the plurality of mipmap images includes, for each mipmap image, assigning a mipmap blur level to the mipmap image, wherein the respective focal plane associated with the particular pixel is determined based on the depth of the particular pixel and based on the mipmap blur level assigned to the mipmap image such that respective focal planes associated with the pixels of the mipmap image vary among different pixels of the mipmap image. ### Claim 15 (depends on 14) The computer-implemented method of claim 14 wherein determining blurred output pixel values of the output image using the at least one selected mipmap image includes: determining a difference between the depth of the designated pixel and the output focal plane depth; and selecting one or more particular mipmap images of the plurality of mipmap images which have a respective mipmap blur level that is closest among the plurality of mipmap images to a lookup value determined based on the depth of the designated pixel and the output focal plane depth. ### Claim 16 (depends on 12) The computer-implemented method of claim 12 further comprising: obtaining a blur strength parameter that indicates a level of blur for the output image, wherein the blur strength parameter is based on user input, wherein determining the blurred output pixel values includes, for each designated pixel of the input image, interpolating between a corresponding mipmap pixel value and the pixel value of the input image using the blur strength parameter. ### Claim 17 (depends on 14) The computer-implemented method of claim 14 wherein the plurality of mipmap images are background mipmap images, wherein applying the blur includes applying a background blur based on pixels associated with a depth behind the respective focal plane associated with the particular pixel, and further comprising: generating a plurality of foreground mipmap images based on the input image, wherein generating each of the foreground mipmap images includes applying a foreground blur to each particular foreground pixel of a second plurality of pixels of the foreground mipmap image, wherein determining the blurred output pixel values includes: determining, based on the output focal plane depth, whether each pixel of the input image is one of a background pixel associated with a depth value behind the focal plane of the output image, or a foreground pixel associated with a depth value in front of the focal plane of the output image; and using at least one of the background mipmap images in response to the pixel of the input image being a background pixel, and using at least one of the foreground mipmap images in response to the pixel of the input image being a foreground pixel. ### Claim 18 (depends on 12) The computer-implemented method of claim 12, wherein applying the blur is applying a background blur based on the particular pixel associated with a depth behind the respective focal plane associated with the particular pixel, and further comprising: obtaining a focal range parameter for the input image, wherein the focal range parameter indicates an output focal range in front of the focal plane of the output image, wherein pixels associated with a depth in the output focal range do not contribute to blur of surrounding pixels that surround the pixels associated with the depth in the output focal range; and adding a respective foreground blur to one or more output pixel values of the output image, wherein the respective foreground blur is based at least in part on the focal range parameter. ### Claim 19 (depends on 18) The computer-implemented method of claim 18, further comprising: generating a plurality of foreground mipmap images based on the input image, wherein adding the respective foreground blur to the one or more output pixel values of the output image includes determining overlay pixel values using at least one foreground mipmap image of the plurality of foreground mipmap images, wherein adding the respective foreground blur includes combining the overlay pixel values with corresponding output pixel values of the output image. ### Claim 20 A system comprising: a memory; and at least one processor configured to access the memory and configured to perform operations comprising: obtaining an input image; generating a plurality of background mipmap images based on the input image, wherein generating each of the background mipmap images includes applying a background blur to a plurality of background pixels derived from the input image for a respective background mipmap image, wherein each background pixel is associated with a respective depth behind a respective focal plane associated with the background pixel; generating a plurality of foreground mipmap images based on the input image, wherein generating each of the foreground mipmap images includes applying a foreground blur to a plurality of foreground pixels derived from the input image for a respective foreground mipmap image, wherein each foreground pixel is associated with a respective depth in front of a respective focal plane associated with the foreground pixel; obtaining parameter data indicating an output focal plane depth for an output focal plane of an output image and indicating an output focal range in front of the output focal plane; and generating output pixel values of the output image that include an output blur, wherein the output blur is based on particular pixels associated with a depth outside the output focal range, and wherein generating output pixel values includes: determining opaque pixel values based on at least one background mipmap image selected based on the output focal plane depth; determining overlay pixel values based on at least one foreground mipmap image selected based on the output focal range; and combining the opaque pixel values with corresponding values of the overlay pixel values to determine the output pixel values of the output image.