# US 2022/0070389 A1 · Matching foreground and virtual background during a video communication session

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0070389 A1
- Document ID: US20220070389A1
- Status: Published application
- Kind: A1
- Application number: 17/010,333
- Filed: 2020-09-02
- Published: 2022-03-03
- Earliest claimed: 2020-09-02
- Assignees: Cisco Technology, Inc.
- Applicants: Cisco Technology, Inc.
- Inventors: Kristian Tangeland; Frode Tjøntveit; Erik Hellerud; Torgeir Moringen; Mattias Ahnoff
- Web page: https://patentgrep.com/patent/US20220070389A1

## CPC

- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- G06T 3/60: Rotation of whole images or parts thereof
- G06T 5/70: Denoising; Smoothing
- G06T 5/73: Deblurring; Sharpening
- G06T 5/90: Dynamic range modification of images or parts thereof
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- H04N 7/15: Conference systems
- H04N 23/73: by influencing the exposure time
- H04N 23/80: Camera processing pipelines; Components thereof
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method, computer system, and computer program product are provided for virtual background replacement during a video communication session. A frame comprising a user image and a captured background is obtained from a video capture device as part of a video stream. The frame is processed to replace the captured background with a virtual background. A change to an exposure characteristic of the video capture device is determined. In response to determining the change to the exposure characteristic, the virtual background is modified to match a virtual background brightness with a user image brightness to produce a modified virtual background. The frame is modified by combining the user image with the modified virtual background.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining, from a video capture device as part of a video stream, a frame comprising a user image and a captured background;
processing the frame to replace the captured background with a virtual background;
determining a change, during the video stream, to an exposure characteristic of the video capture device, wherein the exposure characteristic corresponds to an amount of light entering the video capture device;
in response to determining the change to the exposure characteristic, modifying the virtual background to match a virtual background brightness with a user image brightness to produce a modified virtual background, wherein modifying the virtual background further comprises:
determining one or more user image values including a dynamic range value, a sharpness value, a saturation value, and a noise value; and
applying one or more adjustments to match the virtual background to the user image based on the one or more user image values; and
modifying the frame by combining the user image with the modified virtual background.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
in response to determining the change to the exposure characteristic, modifying the user image brightness in combination with the modifying of the virtual background to match the virtual background brightness and the user image brightness.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein modifying the virtual background further comprises:
determining a virtual background white point value and a user image white point value; and
applying a color correction matrix to match the virtual background white point value and the user image white point value.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein modifying the virtual background further comprises:
determining a virtual background tilt angle and a video capture device tilt angle; and
applying a rotation to the virtual background based on a difference between the video capture device tilt angle and the virtual background tilt angle.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining a video capture device focal length and a virtual background focal length; and
applying a zoom to the virtual background to match the video capture device focal length to the virtual background focal length.

### Claim 6

(canceled)

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the virtual background is recommended to a user based on one or more of: a focal length of the video capture device, an appearance of the user, a time of day, and a location of the user.

### Claim 8

A computer system comprising:
one or more computer processors;
one or more computer readable storage media;
program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising instructions to:
obtain, from a video capture device as part of a video stream, a frame comprising a user image and a captured background;
process the frame to replace the captured background with a virtual background;
determine a change, during the video stream, to an exposure characteristic of the video capture device, wherein the exposure characteristic corresponds to an amount of light entering the video capture device;
in response to determining the change to the exposure characteristic, modify the virtual background to match a virtual background brightness with a user image brightness to produce a modified virtual background, wherein modifying the virtual background further comprises:
determining one or more user image values including a dynamic range value, a sharpness value, a saturation value, and a noise value; and
applying one or more adjustments to match the virtual background to the user image based on the one or more user image values; and
modify the frame by combining the user image with the modified virtual background.

### Claim 9 (depends on 8)

The computer system of claim 8, wherein the program instructions further comprise instructions to:
in response to determining the change to the exposure characteristic, modify the user image brightness in combination with the modifying of the virtual background to match the virtual background brightness and the user image brightness.

### Claim 10 (depends on 8)

The computer system of claim 8, wherein the instructions to modify the virtual background further comprise instructions to:
determine a virtual background white point value and a user image white point value; and
apply a color correction matrix to match the virtual background white point value and the user image white point value.

### Claim 11 (depends on 8)

The computer system of claim 8, wherein the instructions to modify the virtual background further comprise instructions to:
determine a virtual background tilt angle and a video capture device tilt angle; and
apply a rotation to the virtual background based on a difference between the video capture device tilt angle and the virtual background tilt angle.

### Claim 12 (depends on 11)

The computer system of claim 11, further comprising instructions to:
determine a video capture device focal length and a virtual background focal length; and
apply a zoom to the virtual background to match the video capture device focal length to the virtual background focal length.

### Claim 13

(canceled)

### Claim 14 (depends on 8)

The computer system of claim 8, wherein the virtual background is recommended to a user based on one or more of: a focal length of the video capture device, an appearance of the user, a time of day, and a location of the user.

### Claim 15

A computer program product comprising one or more computer readable storage media collectively having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
obtain, from a video capture device as part of a video stream, a frame comprising a user image and a captured background;
process the frame to replace the captured background with a virtual background;
determine a change, during the video stream, to an exposure characteristic of the video capture device, wherein the exposure characteristic corresponds to an amount of light entering the video capture device;
in response to determining the change to the exposure characteristic, modify the virtual background to match a virtual background brightness with a user image brightness to produce a modified virtual background, wherein modifying the virtual background further comprises:
determining one or more user image values including a dynamic range value, a sharpness value, a saturation value, and a noise value; and
applying one or more adjustments to match the virtual background to the user image based on the one or more user image values; and
modify the frame by combining the user image with the modified virtual background.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the program instructions further cause the computer to:
in response to determining the change to the exposure characteristic, modify the user image brightness in combination with the modifying of the virtual background to match the virtual background brightness and the user image brightness.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein the instructions to modify the virtual background further cause the computer to:
determine a virtual background white point value and a user image white point value; and
apply a color correction matrix to match the virtual background white point value and the user image white point value.

### Claim 18 (depends on 15)

The computer program product of claim 15, wherein the instructions to modify the virtual background further cause the computer to:
determine a virtual background tilt angle and a video capture device tilt angle; and
apply a rotation to the virtual background based on a difference between the video capture device tilt angle and the virtual background tilt angle.

### Claim 19 (depends on 18)

The computer program product of claim 18, further comprising instructions to cause the computer to:
determine a video capture device focal length and a virtual background focal length; and
apply a zoom to the virtual background to match the video capture device focal length to the virtual background focal length.

### Claim 20

(canceled)

### Claim 21 (depends on 1)

The computer-implemented method of claim 1, wherein the virtual background comprises a dynamic virtual background.

### Claim 22 (depends on 8)

The computer system of claim 8, wherein the virtual background comprises a dynamic virtual background.

### Claim 23 (depends on 15)

The computer program product of claim 15, wherein the virtual background comprises a dynamic virtual background.
