# US 10,122,965 B2 · Face detection for background management

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,122,965 B2
- Document ID: US10122965B2
- Status: Granted
- Kind: B2
- Application number: 15/364,191
- Filed: 2016-11-29
- Published: 2018-05-31
- Granted: 2018-11-06
- Earliest claimed: 2016-11-29
- Assignees: Facebook, Inc.
- Applicants: Facebook, Inc.
- Inventors: Stephane Taine; Brendan Benjamin Aronoff; Jason Clark
- Web page: https://patentgrep.com/patent/US10122965B2

## CPC

- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- G06T 7/11: Region-based segmentation
- G06T 7/33: using feature-based methods
- G06T 7/73: using feature-based methods
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- H04N 7/15: Conference systems
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30201: Face
- G06T 2210/12: Bounding box
- G06T 2210/22: Cropping
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Exemplary embodiments relate to uses of face detection for modifying the background of a video. By detecting the portion of the video shot that corresponds to a face, it can also determined which portion of the shot do not correspond to the face (e.g., the background of the video). The thus-detected background may be replaced with a movie, event, or other material. Different types of materials may be targeted at different users based on user affinities or preferences. In some embodiments, a user may manually select and apply a background during a video call. When applying a custom background, the system can transmit a control signal and the background. The client recognizes the control signal and, in response, applies the background.

## Claims

### Claim 1

A method, comprising:
accessing one or more frames from a video call, the video call comprising image data and audio data, the one or more frames comprising a face;
performing face detection to detect a portion of the one or more frames that substantially comprises the face;
based on the face detection, assigning an area of the one or more frames not belonging to the face as a background of the one or more frames;
accessing a video channel to a server associated with the video call, accessing an audio channel to the server associated with the video call, and accessing a control channel to the server, the control channel for carrying control data and distinct from the video channel and the audio channel;
forwarding the image data to the server on the video channel and forwarding the audio data to the server on the audio channel; and
transmitting a control signal to the server on the control channel distinct from the video channel and the audio channel, the control signal comprising an instruction to modify the background, an identification of the one or more frames subject to modification, and at least one of a copy of the modified background or a background identifier to a server associated with the video call, the transmitting occurring in a control channel distinct from a channel for transmitting audio and distinct from a channel for transmitting video.

### Claim 2 (depends on 1)

The method of claim 1, wherein modifying the background comprises replacing the background with a new background.

### Claim 3 (depends on 1)

The method of claim 1, wherein the modified background comprises material selected based on one or more characteristics associated with a user whose face appears in the one or more frames.

### Claim 4 (depends on 1)

The method of claim 1, wherein the modified background includes variable promotional material.

### Claim 5 (depends on 1)

The method of claim 1, wherein modifying the background is performed in response to receiving a background modification instruction in an interface of the video call.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
receiving a video signal comprising video from a sender client, a control signal associated with the video signal, and a copy of a background or a background identifier associated with the control signal; and
in response to receiving the control signal, performing sender face detection on the video signal to identify a sender face in the video signal;
based on the sender face detection, assigning an area of the one or more frames not belonging to the sender face as a sender background of the video signal; and
replacing the sender background with the copy of the background or with a local copy of a background associated with the background identifier.

### Claim 7

A non-transitory computer-readable medium storing instructions configured to cause one or more processors to:
access one or more frames from a video call, the video call comprising image data and audio data, the one or more frames comprising a face;
perform face detection to detect a portion of the one or more frames that substantially comprises the face;
based on the face detection, assign an area of the one or more frames not belonging to the face as a background of the one or more frames;
access a video channel to a server associated with the video call, accessing an audio channel to the server associated with the video call, and accessing a control channel to the server, the control channel for carrying control data and distinct from the video channel and the audio channel;
forward the image data to the server on the video channel and forwarding the audio data to the server on the audio channel; and
transmit a control signal to the server on the control channel distinct from the video channel and the audio channel, the control signal comprising an instruction to modify the background, an identification of the one or more frames subject to modification, and at least one of a copy of the modified background or a background identifier to a server associated with the video call, the transmitting occurring in a control channel distinct from a channel for transmitting audio and distinct from a channel for transmitting video.

### Claim 8 (depends on 7)

The medium of claim 7, wherein modifying the background comprises replacing the background with a new background.

### Claim 9 (depends on 7)

The medium of claim 7, wherein the modified background comprises material selected based on one or more characteristics associated with a user whose face appears in the one or more frames.

### Claim 10 (depends on 7)

The medium of claim 7, wherein the modified background includes variable promotional material.

### Claim 11 (depends on 7)

The medium of claim 7, wherein modifying the background is performed in response to receiving a background modification instruction in an interface of the video call.

### Claim 12 (depends on 7)

The medium of claim 7, further storing instructions for:
receiving a video signal comprising video from a sender client, a control signal associated with the video signal, and a copy of a background or a background identifier associated with the control signal; and
in response to receiving the control signal, performing sender face detection on the video signal to identify a sender face in the video signal;
based on the sender face detection, assigning an area of the one or more frames not belonging to the sender face as a sender background of the video signal; and
replacing the sender background with the copy of the background or with a local copy of a background associated with the background identifier.

### Claim 13

An apparatus comprising:
a non-transitory computer readable medium configured to store instructions for facilitating a video call; and
a processor configured to execute the instructions, the instructions configured to cause the processor to:
access one or more frames from a video call, the video call comprising image data and audio data, the one or more frames comprising a face;
perform face detection to detect a portion of the one or more frames that substantially comprises the face;
based on the face detection, assign an area of the one or more frames not belonging to the face as a background of the one or more frames;
access a video channel to a server associated with the video call, accessing an audio channel to the server associated with the video call, and accessing a control channel to the server, the control channel for carrying control data and distinct from the video channel and the audio channel;
forward the image data to the server on the video channel and forwarding the audio data to the server on the audio channel; and
transmit a control signal to the server on the control channel distinct from the video channel and the audio channel, the control signal comprising an instruction to modify the background, an identification of the one or more frames subject to modification, and at least one of a copy of the modified background or a background identifier to a server associated with the video call, the transmitting occurring in a control channel distinct from a channel for transmitting audio and distinct from a channel for transmitting video.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein modifying the background comprises replacing the background with a new background.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the modified background comprises material selected based on one or more characteristics associated with a user whose face appears in the one or more frames.

### Claim 16 (depends on 13)

The apparatus of claim 13, wherein the modified background includes variable promotional material.

### Claim 17 (depends on 13)

The apparatus of claim 13, wherein modifying the background is performed in response to receiving a background modification instruction in an interface of the video call.