# US 11,451,758 B1 · Systems, methods, and media for colorizing grayscale images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,451,758 B1
- Document ID: US11451758B1
- Status: Granted
- Kind: B1
- Application number: 16/789,019
- Filed: 2020-02-12
- Granted: 2022-09-20
- Earliest claimed: 2020-02-12
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Gaurav Chaurasia; Alexander Sorkine Hornung; David Novotny; Nikola Dodik
- Web page: https://patentgrep.com/patent/US11451758B1

## CPC

- G06T 5/77: Retouching; Inpainting; Scratch removal
- H04N 13/15: for colour aspects of image signals
- G06N 20/00: Machine learning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 7/90: Determination of colour characteristics
- H04N 13/133: Equalising the characteristics of different image components, e.g. their average brightness or colour balance
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/344: with head-mounted left-right displays
- G06T 2207/10012: Stereo images
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- H04N 2013/0077: {Colour aspects}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In one embodiment, a computing system may access a first grayscale image and a second grayscale image. The system may generate a first color image and a second color image based on the first grayscale image and the second grayscale image, respectively. The system may generate affinity information based on the first grayscale image and the second grayscale image, the affinity information identifying relationships between pixels of the first grayscale image and pixels of the second grayscale image. The system may modify the color of the first color image and the second color image based on the affinity information. The system may generate a first visual output based on the modified first color image and a second visual output based on the modified second color image.

## Claims

### Claim 1

A method comprising, by a computing system:
capturing a first grayscale image by a first camera of a device and a second grayscale image by a second camera of the device, wherein the first and second grayscale images are stereo images that are captured at a same time;
accessing the first grayscale image and the second grayscale image that are captured by the first camera and second camera, respectively;
generating, using a machine-learning model, a first color image corresponding to the first grayscale image and a second color image corresponding to the second grayscale image, wherein the first and second color images include color inconsistencies;
generating affinity information, for resolving the color inconsistencies in the first and second color images, based on comparison of pixels between the first grayscale image and the second grayscale image, the affinity information identifying relationships between pixels of the first grayscale image and pixels of the second grayscale image;
modifying the first color image and the second color image, which are generated using the machine-learning model, based on the relationships between the pixels of the first grayscale image and the pixels of the second grayscale image, wherein the modification resolves the color inconsistencies in the first color image and the second color image; and
generating a first visual output based on the modified first color image and a second visual output based on the modified second color image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the machine-learning model is trained to colorize grayscale images, and wherein the machine-learning is trained based on a set of training samples, each training sample comprising a particular grayscale image and a corresponding ground-truth colored image of the particular grayscale image.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
an object appears in the first color image and the second color image, wherein the object's color is inconsistent between the first color image and the second color image prior to the modification; and
the object's color between the modified first color image and the modified second color image is more consistent than the object's color between the first color image and the second color image prior to the modification.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
an object appears in the first grayscale image, the first color image, the second grayscale image, and the second color image; and
the object's color in the modified first color image and the object's color in the modified second color image are more similar than the object's color in the first color image and the object's color in the second color image.

### Claim 5 (depends on 4)

The method of claim 4, wherein the affinity information indicates that a first collection of pixels in the first grayscale image corresponding to the object is related to a second collection of pixels in the second grayscale image corresponding to the object.

### Claim 6 (depends on 1)

The method of claim 1, wherein the affinity information indicates that a first collection of pixels in the first grayscale image are related.

### Claim 7 (depends on 6)

The method of claim 6, wherein:
the first grayscale image and the second grayscale image are simultaneously captured by, respectively, the first camera and the second camera having an overlapping field of view; and
the affinity information indicates that the first collection of pixels in the first grayscale image is related to a second collection of pixels in the second grayscale image.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
generating a third color image based on a third grayscale image, the third grayscale image and the first grayscale image being captured sequentially by the first camera;
modifying color of the third color image based on the affinity information; and
outputting the modified third color image;
wherein generating the affinity information is further based on the third grayscale image; and
wherein the affinity information indicates that the first collection of pixels in the first grayscale image is related to a third collection of pixels in the third grayscale image.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
displaying the first visual output and the second visual output simultaneously on a stereo display.

### Claim 10 (depends on 1)

The method of claim 1, wherein the device is a head-mounted display device.

### Claim 11

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
capture a first grayscale image by a first camera of a device and a second grayscale image by a second camera of the device, wherein the first and second grayscale images are stereo images that are captured at a same time;
access the first grayscale image and the second grayscale image that are captured by the first camera and second camera, respectively;
generate, using a machine-learning model, a first color image corresponding to the first grayscale image and a second color image corresponding to the second grayscale image, wherein the first and second color images include color inconsistencies;
generate affinity information, for resolving the color inconsistencies in the first and second color images, based on comparison of pixels between the first grayscale image and the second grayscale image, the affinity information identifying relationships between pixels of the first grayscale image and pixels of the second grayscale image;
modify the first color image and the second color image, which are generated using the machine-learning model, based on the relationships between the pixels of the first grayscale image and the pixels of the second grayscale image, wherein the modification resolves the color inconsistencies in the first color image and the second color image; and
generate a first visual output based on the modified first color image and a second visual output based on the modified second color image.

### Claim 12 (depends on 11)

The media of claim 11, wherein:
an object appears in the first grayscale image, the first color image, the second grayscale image, and the second color image; and
the object's color in the modified first color image and the object's color in the modified second color image are more similar than the object's color in the first color image and the object's color in the second color image.

### Claim 13 (depends on 11)

The media of claim 11, wherein the affinity information indicates that a first collection of pixels in the first grayscale image are related.

### Claim 14 (depends on 13)

The media of claim 13, wherein:
the first grayscale image and the second grayscale image are simultaneously captured by, respectively, the first camera and the second camera having an overlapping field of view; and
the affinity information indicates that the first collection of pixels in the first grayscale image is related to a second collection of pixels in the second grayscale image.

### Claim 15 (depends on 14)

The media of claim 14, wherein the software is further operable when executed to:
generate a third color image based on a third grayscale image, the third grayscale image and the first grayscale image being captured sequentially by the first camera;
modify color of the third color image based on the affinity information; and
output the modified third color image;
wherein generating the affinity information is further based on the third grayscale image; and
wherein the affinity information indicates that the first collection of pixels in the first grayscale image is related to a third collection of pixels in the third grayscale image.

### Claim 16

A system comprising:
one or more processors; and
one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
capture a first grayscale image by a first camera of a device and a second grayscale image by a second camera of the device, wherein the first and second grayscale images are stereo images that are captured at a same time;
access the first grayscale image and the second grayscale image that are captured by the first camera and second camera, respectively;
generate, using a machine-learning model, a first color image corresponding to the first grayscale image and a second color image corresponding to the second grayscale image, wherein the first and second color images include color inconsistencies;
generate affinity information, for resolving the color inconsistencies in the first and second color images, based on comparison of pixels between the first grayscale image and the second grayscale image, the affinity information identifying relationships between pixels of the first grayscale image and pixels of the second grayscale image;
modify the first color image and the second color image, which are generated using the machine-learning model, based on the relationships between the pixels of the first grayscale image and the pixels of the second grayscale image, wherein the modification resolves the color inconsistencies in the first color image and the second color image; and
generate a first visual output based on the modified first color image and a second visual output based on the modified second color image.

### Claim 17 (depends on 16)

The system of claim 16, wherein:
an object appears in the first grayscale image, the first color image, the second grayscale image, and the second color image; and
the object's color in the modified first color image and the object's color in the modified second color image are more similar than the object's color in the first color image and the object's color in the second color image.

### Claim 18 (depends on 16)

The system of claim 16, wherein the affinity information indicates that a first collection of pixels in the first grayscale image are related.

### Claim 19 (depends on 18)

The system of claim 18, wherein:
the first grayscale image and the second grayscale image are simultaneously captured by, respectively, the first camera and the second camera having an overlapping field of view; and
the affinity information indicates that the first collection of pixels in the first grayscale image is related to a second collection of pixels in the second grayscale image.

### Claim 20 (depends on 19)

The system of claim 19, wherein the one or more of the processors are further operable when executing the instructions to cause the system to:
generate a third color image based on a third grayscale image, the third grayscale image and the first grayscale image being captured sequentially by the first camera;
modify color of the third color image based on the affinity information; and
output the modified third color image;
wherein generating the affinity information is further based on the third grayscale image; and
wherein the affinity information indicates that the first collection of pixels in the first grayscale image is related to a third collection of pixels in the third grayscale image.