# US 2013/0279588 A1 · Using Depth Information to Assist Motion Compensation-Based Video Coding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0279588 A1
- Document ID: US20130279588A1
- Status: Published application
- Kind: A1
- Application number: 13/864,672
- Filed: 2013-04-17
- Published: 2013-10-24
- Earliest claimed: 2012-04-19
- Assignees: Futurewei Technologies, Inc.
- Applicants: Futurewei Technologies, Inc.
- Inventors: Zhenyu Wu; Hong Heather Yu
- Web page: https://patentgrep.com/patent/US20130279588A1

## CPC

- H04N 19/513: Processing of motion vectors
- H04N 19/23: with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
- H04N 19/43: Hardware specially adapted for motion estimation or compensation
- H04N 19/54: using feature points or meshes
- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N 19/527: Global motion vector estimation
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An apparatus comprising a plurality of ports, and a processor coupled to the ports and configured to receive a plurality of video frames from any of the ports, wherein the video frames comprise an object and a background, and wherein the video frames comprise a plurality of color pixels for the object and the background, receive a plurality of depth frames from any of the ports, wherein the depth frames comprise the object and the background, and wherein the depth frames comprise an indication of an object depth relative to a background depth, and encode the video frames using the indication of the object depth relative to the background depth.

## Claims

### Claim 1

An apparatus comprising:
a plurality of ports; and
a processor coupled to the ports and configured to:
receive a plurality of video frames from any of the ports, wherein the video frames comprise an object and a background, and wherein the video frames comprise a plurality of color pixels for the object and the background;
receive a plurality of depth frames from any of the ports, wherein the depth frames comprise the object and the background, and wherein the depth frames comprise an indication of an object depth relative to a background depth; and
encode the video frames using the indication of the object depth relative to the background depth.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein encoding the video frame comprises:
extracting the color pixels from the video frames;
extracting the indication of an object depth relative to a background depth from the depth frames;
segmenting the object from the background using the indication of the object depth relative to the background depth;
calculating a motion of the object; and
encoding the object.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein calculating the motion of the object comprises:
determining that the object is not rigid; and
performing a motion estimation for the object.

### Claim 4 (depends on 2)

The apparatus of claim 2, wherein calculating the motion of the object comprises:
determining that the object is rigid;
tracking the object across the background over the frames;
modeling the motion of the object; and
deriving a motion vector for a first coding block of the object.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the motion vector is used to describe a motion of the first coding block, but motion estimation is used to describe a motion of a second coding block of the video frames.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the modeling is based on a translational, geometric, affine, or perspective model.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the indication of the object depth relative to the background depth is represented by differences in color intensity.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the indication of the object depth relative to the background depth is represented by numerical distances of the object from a depth sensor.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the video frames are used for videoconferencing, telepresence, or video surveillance.

### Claim 10

A system comprising:
a camera configured to obtain a plurality of video frames, wherein the video frames comprise an object and a background, and wherein the video frames comprise a plurality of color pixels for the object and the background;
a depth device configured to obtain a plurality of depth frames, wherein the depth frames comprise the object and the background, and wherein the depth frames comprise an indication of an object depth relative to a background depth; and
an encoder coupled to the camera and the depth device, wherein the encoder is configured to encode the video frames using the indication of the object depth relative to the background depth.

### Claim 11 (depends on 10)

The system of claim 10, wherein the encoder comprises:
an image extraction sub-module configured to extract the color pixels from the video frames and extract the indication of an object depth relative to a background depth from the depth frames;
an object segmentation sub-module configured to segment the object from the background using the indication of the object depth relative to the background depth;
a motion calculation sub-module configured to calculate a motion of the object; and
a video encoding sub-module configured to encode the object.

### Claim 12 (depends on 11)

The system of claim 11, wherein calculating the motion of the object comprises:
determining whether the object is rigid;
performing a motion estimation for the object when the object is not rigid;
tracking the object across the background over the frames when the object is rigid;
modeling the motion of the object when the object is rigid; and
deriving a motion vector for a coding block of the object when the object is rigid.

### Claim 13 (depends on 12)

The system of claim 12, wherein the depth device is an infrared camera.

### Claim 14 (depends on 12)

The system of claim 12, wherein the depth device is an ultraviolet camera.

### Claim 15 (depends on 12)

The system of claim 12, wherein the depth device is a radar system or a sonar system.

### Claim 16 (depends on 13)

The system of claim 13, wherein the depth device is a depth sensor, a light field camera, a ranging camera, a complementary metal-oxide-semiconductor (CMOS) visual sensor, a charged-coupled device (CCD) visual sensor, or a combination of visual sensors.

### Claim 17

A method comprising:
receiving a plurality of video frames from any of the ports, wherein the video frames comprise an object and a background, and wherein the video frames comprise a plurality of color pixels for the object and the background;
receiving a plurality of depth frames from any of the ports, wherein the depth frames comprise the object and the background, and wherein the depth frames comprise an indication of an object depth relative to a background depth; and
encoding the video frames using the indication of the object depth relative to the background depth.

### Claim 18 (depends on 17)

The method of claim 17, wherein encoding the video frame comprises:
extracting the color pixels from the video frames;
extracting the indication of an object depth relative to a background depth from the depth frames;
segmenting the object from the background using the indication of the object depth relative to the background depth;
calculating a motion of the object; and
encoding the object.

### Claim 19 (depends on 18)

The method of claim 18, wherein calculating the motion of the object comprises:
determining that the object is not rigid; and
performing a motion estimation for the object.

### Claim 20 (depends on 18)

The method of claim 18, wherein estimating the motion of the object comprises:
determining that the object is rigid;
tracking the object across the background over the frames;
modeling the motion of the object; and
deriving a motion vector for a coding block of the object.
