# US 12,008,766 B2 · Image processing to detect edges, walls, and surfaces for a virtual painting application

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,766 B2
- Document ID: US12008766B2
- Status: Granted
- Kind: B2
- Application number: 18/225,714
- Filed: 2023-07-25
- Published: 2024-02-01
- Granted: 2024-06-11
- Earliest claimed: 2022-07-26
- Assignees: Behr Process Corporation
- Applicants: Behr Process Corporation
- Inventors: Douglas Milsom; Un Ho Chung; Michael Askew; Kiki Takakura-Merquise
- Web page: https://patentgrep.com/patent/US12008766B2

## CPC

- G06T 7/13: Edge detection
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/194: involving foreground-background segmentation
- G06T 7/50: Depth or shape recovery
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/46: Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56)
- G06V 10/56: relating to colour
- G06V 20/36: {Indoor scenes}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 2207/20044: Skeletonization; Medial axis transform
- G06T 2207/20132: Image cropping
- G06T 2207/30168: Image quality inspection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for enabling a user to paint an image uploaded to a device is provided. The method includes uploading an image to the device, performing a depth estimation on the image to generate a depth map, and performing an edge detection on the image to generate a first edge map. The method also includes performing the edge detection on the depth map to generate a second edge map, performing a skeletonization function on the first edge map to generate a third edge map, and performing the skeletonization function on the depth map to generate a fourth edge map. The method also includes generating a final edge map using the third edge map and the fourth edge map, generating a colorized image by applying color to the final edge map to paint at least one wall of a scene, and displaying the colorized image on a display of the device.

## Claims

### Claim 1

A method for enabling a user to paint an image uploaded to a device, the method comprising:
uploading an image to the device, wherein the image is of a scene having at least one wall;
performing a depth estimation on the image to generate a depth map;
performing an edge detection on the image to generate a first edge map, wherein the first edge map includes a first set of edge lines of the scene outlining the at least one wall;
performing the edge detection on the depth map to generate a second edge map, wherein the first edge map includes a second set of edge lines of the scene;
performing a skeletonization function on the first edge map to generate a third edge map, wherein the skeletonization function thins a width of each edge line in the first set of edge lines;
performing the skeletonization function on the depth map to generate a fourth edge map;
generating a final edge map using the third edge map and the fourth edge map;
generating a colorized image by applying color to the final edge map to paint the at least one wall; and
displaying the colorized image on a display of the device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising validating the image before performing the depth estimation, wherein the validating includes determining whether the image meets a threshold for image quality.

### Claim 3 (depends on 1)

The method of claim 1, further comprising adjusting the image before performing the depth estimation, wherein the adjusting includes at least one of cropping the image, modifying an angle of the image, zooming in or out of the image, and modifying a placement of the image.

### Claim 4 (depends on 1)

The method of claim 1, further comprising scaling the image before performing the depth estimation, wherein the scaling includes adjusting a resolution of the image to a maximum resolution.

### Claim 5 (depends on 4)

The method of claim 4, wherein the maximum resolution is one of 1600 pixels by 900 pixels or 900 pixels by 1600 pixels.

### Claim 6 (depends on 1)

The method of claim 1, wherein performing the edge detection includes performing a raw holistically-nested edge detection (HED).

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
performing a canny edge detection on the image to generate a fifth edge map;
performing the canny edge detection on a saturation channel of the image to generate a sixth edge map; and
wherein the generating the final edge map is using the third, fourth, fifth and sixth edge maps.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
down-sampling the image to generate a down-sampled image;
performing a canny edge detection on the down-sampled image to generate an edge map;
up-sampling the edge map to generate an up-sampled edge map;
performing the skeletonization function on the up-sampled edge map to generate a seventh edge map; and
wherein the generating the final edge map is using the third, fourth, and seventh edge maps.

### Claim 9

A method for enabling a user to paint an image uploaded to a device, the method comprising:
uploading an image to the device, wherein the image is of a scene having at least one wall;
performing a depth estimation on the image to generate a depth map;
performing a first edge detection on the image to generate a first edge map, wherein the first edge map includes a first set of edge lines of the scene outlining the at least one wall;
performing the first edge detection on the depth map to generate a second edge map, wherein the second edge map includes a second set of edge lines of the scene;
performing a skeletonization function on the first edge map to generate a third edge map, wherein the skeletonization function thins a width of each edge line in the first set of edge lines;
performing the skeletonization function on the depth map to generate a fourth edge map;
performing a second edge detection on the image to generate a fifth edge map;
performing the second edge detection on a saturation channel of the image to generate a sixth edge map;
generating a final edge map using the third edge map, the fourth edge map, the fifth edge map, and the sixth edge maps;
generating a colorized image by applying color to the final edge map to paint the at least one wall; and
displaying the colorized image on a display of the device.

### Claim 10 (depends on 9)

The method of claim 9, wherein performing the first edge detection includes performing a raw holistically-nested edge detection (HED).

### Claim 11 (depends on 9)

The method of claim 9, wherein performing the second edge detection includes performing a canny edge detection.

### Claim 12 (depends on 9)

The method claim 9, further comprising measuring a first value of a brightness and a second value of a contrast of the image, and balancing a color of the image using the first and second values.

### Claim 13 (depends on 9)

The method of claim 9, further comprising:
down-sampling the image to generate a down-sampled image;
performing a canny edge detection on the down-sampled image to generate an edge map;
up-sampling the edge map to generate an up-sampled edge map;
performing the skeletonization function on the up-sampled edge map to generate a seventh edge map; and
wherein the generating the final edge map is using the third, fourth, fifth, sixth, and seventh edge maps.

### Claim 14

A method for enabling a user to paint an image uploaded to a device, the method comprising:
uploading an image to the device, wherein the image is of a scene having at least one wall, and the scene includes at least one shadow and at least one highlight;
performing a depth estimation on the image to generate a depth map;
performing an edge detection on the image to generate an edge map, wherein the edge map includes an outline of the at least one wall;
generating a final edge map using the depth map and the edge map;
performing a grayscale function on the image to generate a grayscale image;
performing a transparency function on the image to generate an alpha gray image, wherein the alpha gray image represents the at least one shadow as an opaque region and represents the at least one highlight as a transparent region;
performing a normalization function on the image to generate a re-gray image, wherein the normalization function includes measuring one or more values of at least one pixel of the image and normalizing the one or more values;
generating a colorized image by applying a color to the final edge map and using the grayscale image, alpha gray image and re-gray image; and
displaying the colorized image on a display of the device.

### Claim 15 (depends on 14)

The method of claim 14, wherein performing the grayscale function includes gray scaling the image to generate a preliminary grayscale image and adjusting a value of at least one pixel of the preliminary grayscale image to generate the grayscale image.

### Claim 16 (depends on 14)

The method of claim 14, wherein the normalization function further includes determining a dominant color value of the image and the value includes a color value, wherein normalizing the color values of image uses the dominant color value.

### Claim 17 (depends on 14)

The method of claim 14, wherein the normalization function further includes detecting a contour in the image, and the at least one pixel represents pixels of the contour.

### Claim 18 (depends on 17)

The method of claim 17, wherein the one or more values include a brightness value and a contrast value.

### Claim 19 (depends on 14)

The method of claim 14, wherein generating the colorized image includes applying a color on top of the alpha gray image.

### Claim 20 (depends on 14)

The method of claim 14, wherein generating the colorized image includes applying a color below the alpha gray image.

### Claim 21

A system comprising a computing device having a processor and a memory configured to:
receive an image uploaded to the computing device, wherein the image is of a scene having at least one wall,
perform a depth estimation on the image to generate a depth map,
perform an edge detection on the image to generate a first edge map, wherein the first edge map includes a first set of edge lines of the scene outlining the at least one wall,
perform the edge detection on the depth map to generate a second edge map, wherein the first edge map includes a second set of edge lines of the scene,
perform a skeletonization function on the first edge map to generate a third edge map, wherein the skeletonization function thins a width of each edge line in the first set of edge lines,
perform the skeletonization function on the depth map to generate a fourth edge map,
generate a final edge map using the third edge map and the fourth edge map,
generate a colorized image by applying color to the final edge map to paint the at least one wall, and
display the colorized image on a display of the computing device.

### Claim 22 (depends on 21)

The system of claim 21, wherein the computing device is further configured to validate the image before performing the depth estimation, wherein the validating includes determining whether the image meets a threshold for image quality.

### Claim 23 (depends on 21)

The system of claim 21, the computing device is further configured to adjust the image before performing the depth estimation, wherein the adjusting includes at least one of cropping the image, modifying an angle of the image, zooming in or out of the image, and modifying a placement of the image.

### Claim 24 (depends on 21)

The system of claim 21, the computing device is further configured to scale the image before performing the depth estimation, wherein the scaling includes adjusting a resolution of the image to a maximum resolution.

### Claim 25 (depends on 24)

The system of claim 24, wherein the maximum resolution is one of 1600 pixels by 900 pixels or 900 pixels by 1600 pixels.

### Claim 26 (depends on 21)

The system of claim 21, wherein the computing device is further configured to perform the edge detection by performing a raw holistically-nested edge detection (HED).

### Claim 27 (depends on 21)

The system of claim 21, the computing device is further configured to:
perform a canny edge detection on the image to generate a fifth edge map, and
perform the canny edge detection on a saturation channel of the image to generate a sixth edge map;
wherein generating the final edge map is using the third, fourth, fifth and sixth edge maps.

### Claim 28 (depends on 21)

The system of claim 21, the computing device is further configured to:
down-sample the image to generate a down-sampled image,
perform a canny edge detection on the down-sampled image to generate an edge map,
up-sample the edge map to generate an up-sampled edge map, and
perform the skeletonization function on the up-sampled edge map to generate a seventh edge map;
wherein generating the final edge map is using the third, fourth, and seventh edge maps.

### Claim 29

A system comprising a computing device having a processor and a memory configured to:
receive an image uploaded to the computing device, wherein the image is of a scene having at least one wall,
perform a depth estimation on the image to generate a depth map,
perform a first edge detection on the image to generate a first edge map, wherein the first edge map includes a first set of edge lines of the scene outlining the at least one wall,
perform the first edge detection on the depth map to generate a second edge map, wherein the second edge map includes a second set of edge lines of the scene,
perform a skeletonization function on the first edge map to generate a third edge map, wherein the skeletonization function thins a width of each edge line in the first set of edge lines,
perform the skeletonization function on the depth map to generate a fourth edge map,
perform a second edge detection on the image to generate a fifth edge map,
perform the second edge detection on a saturation channel of the image to generate a sixth edge map,
generate a final edge map using the third edge map, the fourth edge map, the fifth edge map, and the sixth edge maps,
generate a colorized image by applying color to the final edge map to paint the at least one wall, and
display the colorized image on a display of the computing device.

### Claim 30 (depends on 29)

The system of claim 29, wherein the computing device is further configured to perform the first edge detection by performing a raw holistically-nested edge detection (HED).

### Claim 31 (depends on 29)

The system of claim 29, wherein the computing device is further configured to perform the second edge detection by performing a canny edge detection.

### Claim 32 (depends on 29)

The system of claim 29, wherein the computing device is further configured to measure a first value of a brightness and a second value of a contrast of the image, and to balance a color of the image using the first and second values.

### Claim 33 (depends on 29)

The system of claim 29, wherein the computing device is further configured to:
down-sample the image to generate a down-sampled image,
perform a canny edge detection on the down-sampled image to generate an edge map,
up-sample the edge map to generate an up-sampled edge map, and
perform the skeletonization function on the up-sampled edge map to generate a seventh edge map;
wherein generating the final edge map is using the third, fourth, fifth, sixth, and seventh edge maps.

### Claim 34

A system comprising a computing device having a processor and a memory configured to:
receive an image uploaded to the computing device, wherein the image is of a scene having at least one wall, and the scene includes at least one shadow and at least one highlight,
perform a depth estimation on the image to generate a depth map,
perform an edge detection on the image to generate an edge map, wherein the edge map includes an outline of the at least one wall,
generate a final edge map using the depth map and the edge map,
perform a grayscale function on the image to generate a grayscale image,
perform a transparency function on the image to generate an alpha gray image, wherein the alpha gray image represents the at least one shadow as an opaque region and represents the at least one highlight as a transparent region,
perform a normalization function on the image to generate a re-gray image, wherein the normalization function includes measuring one or more values of at least one pixel of the image and normalizing the one or more values,
generate a colorized image by applying a color to the final edge map and using the grayscale image, alpha gray image and re-gray image, and
display the colorized image on a display of the computing device.

### Claim 35 (depends on 34)

The system of claim 34, wherein the computing device is further configured to perform the grayscale function by gray scaling the image to generate a preliminary grayscale image and to adjust a value of at least one pixel of the preliminary grayscale image to generate the grayscale image.

### Claim 36 (depends on 34)

The system of claim 34, wherein the computing device is further configured to perform the normalization function further by determining a dominant color value of the image and the value includes a color value, wherein normalizing the color values of image uses the dominant color value.

### Claim 37 (depends on 34)

The system of claim 34, wherein the computing device is further configured to perform the normalization function further by detecting a contour in the image, and the at least one pixel represents pixels of the contour.

### Claim 38 (depends on 37)

The system of claim 37, wherein the one or more values include a brightness value and a contrast value.

### Claim 39 (depends on 34)

The system of claim 34, wherein the computing device is further configured to generate the colorized image includes by applying a color on top of the alpha gray image.

### Claim 40 (depends on 34)

The system of claim 34, wherein the computing device is further configured to generate the colorized image includes applying a color below the alpha gray image.