# US 10,313,417 B2 · Methods and systems for auto-zoom based adaptive video streaming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,313,417 B2
- Document ID: US10313417B2
- Status: Granted
- Kind: B2
- Application number: 15/433,763
- Filed: 2017-02-15
- Published: 2017-10-19
- Granted: 2019-06-04
- Earliest claimed: 2016-04-18
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Ying Chen; Dashan Gao
- Web page: https://patentgrep.com/patent/US10313417B2

## CPC

- H04L 65/764: {at the destination (reformatting of additional data in video clients H04N21/4355)}
- H04L 65/61: for supporting one-way streaming services, e.g. Internet radio
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/162: User input
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/33: in the spatial domain
- H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks
- H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Automatic adaptive zoom enables computing devices that receive video streams to use a higher resolution stream when the user enables zoom, so that the quality of the output video is preserved. In some examples, a tracking video stream and a target video stream are obtained and are processed. The tracking video stream has a first resolution, and the target video stream has a second resolution that is higher than the first resolution. The tracking video stream is processed to define regions of interest for frames of the tracking video stream. The target video stream is processed to generate zoomed-in regions of frames of the target video stream. A zoomed-in region of the target video stream corresponds to a region of interest defined using the tracking video stream. The zoomed-in regions of the frames of the target video stream are then provided for display on a client device.

## Claims

### Claim 1

A method of processing video data, the method comprising:
outputting a first video stream the first video stream having a first resolution;
receiving input corresponding to selection of a location in a first video frame of the first video stream, wherein the input is associated with a request to zoom in on the location in the first video frame;
determining a region of interest using the first video stream, the region of interest including an area around the location in the first video frame, wherein the region of interest is less than an entire area of the first video frame;
identifying a set of tiles of a second video stream, the set of tiles corresponding to the region of interest, the second video stream having a second resolution, wherein the second resolution is a higher resolution than the first resolution, and wherein the set of tiles include fewer than all tiles within video frames of the second video stream;
enabling receipt of sections of the video frames of the second video stream, the sections of the video frames including the set of tiles and excluding areas of the video frames outside of the set of tiles;
enlarging the sections of the video frames according to a zoom factor; and
outputting the enlarged sections of the video frames.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
enabling receipt of a third video stream, the third video stream having a lower resolution than the second resolution and a higher resolution than the first resolution; and
outputting the third video stream after the input is received and until the enlarged sections of the video frames are ready for rendering.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
enabling receipt of a third video stream, the third video stream having a lower resolution than the second resolution and a higher resolution than the first resolution;
extracting from third video frames from the third video stream sections of the third video frames that correspond to the region of interest; and
outputting the sections of the third video frames after the input is received and until the enlarged sections of the video frames are ready for rendering.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
receiving second input corresponding to a request to enable a normal display mode;
enabling receipt of a third video stream, the third video stream having a lower resolution than the second resolution and a higher resolution than the first resolution; and
outputting the third video stream.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first video stream has a first frame rate and the second video stream has a second frame rate, wherein the second frame rate is a higher frame rate than the first frame rate.

### Claim 6 (depends on 1)

The method of claim 1, wherein the region of interest is described by information including a bounding box and a mapping of the bounding box to the first video frame.

### Claim 7 (depends on 1)

The method of claim 1, wherein the request is for enabling an auto-zoom playing mode, wherein, in the auto-zoom playing mode, the enlarged sections of the video frames track the region of interest as the region of interest moves between the video frames.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
outputting the first video stream after the input is received and until the enlarged sections of the video frames are ready for rendering.

### Claim 9 (depends on 1)

The method of claim 1, wherein the video frames of the second video stream are coded using tile-based video coding, wherein tile-based video coding divides each of the video frames into a plurality of tiles.

### Claim 10 (depends on 9)

The method of claim 9, wherein each tile of the plurality of tiles is arranged in a separate track.

### Claim 11 (depends on 1)

The method of claim 1, wherein the set of tiles are associated with a second video frame from the video frames, and further comprising:
identifying a second set of tiles in a third video frame from the video frames, the second set of tiles corresponding to the region of interest with respect to the third video frame, wherein the second set of tiles are different from the set of tiles, and wherein the enlarged sections of the video frames include the second set of tiles.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
identifying an object at the location;
identifying a second region of interest in a second video frame, the second region of interest include an area around a second location of the object in the second video frame; and
identifying a second set of tiles of the second video stream, the second set of tiles corresponding to the second region of interest, wherein the enlarged sections of the video frames include the second set of tiles.

### Claim 13 (depends on 1)

The method of claim 1, wherein the first resolution is associated with a first bitrate, and wherein the first bitrate corresponds to an available bandwidth for receiving the first video stream.

### Claim 14 (depends on 13)

The method of claim 13, wherein the sections of the video frames are associated with a second bitrate, and wherein the second bitrate also corresponds to the available bandwidth.

### Claim 15

An apparatus comprising:
a memory configured to store video data; and
a processor configured to:
output a first video stream the first video stream having a first resolution;
receive input corresponding to selection of a location in a first video frame of the first video stream, wherein the input is associated with a request to zoom in on the location in the first video frame;
determine a region of interest using the first video stream, the region of interest including an area around the location in the first video frame, wherein the region of interest is less than an entire area of the first video frame;
identify a set of tiles of a second video stream, the set of tiles corresponding to the region of interest, the target the second video stream having a second resolution, wherein the second resolution is a higher resolution than the first resolution, and wherein the set of tiles include fewer than all tiles within video frames of the second video stream;
enable receipt of sections of the video frames of the second video stream, the sections of the video frames including the set of tiles and excluding areas of the video frames outside of the set of tiles;
enlarge the sections of the video frames according to a zoom factor; and
output the enlarged sections of the video frames of the second video stream.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the processor is further configured to:
enable receipt of a third video stream, the third video stream having a lower resolution than the second resolution and a higher resolution than the first resolution; and
output the third video stream after the input is received and until the enlarged sections of the video frames are ready for rendering.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the processor is further configured to:
enable receipt of a third video stream, the third video stream having a lower resolution than the second resolution and a higher resolution than the first resolution;
extract from third video frames from the third video stream sections of the third video frames that correspond to the region of interest; and
output the sections of the third video frames after the input is received and until the enlarged sections of the video frames are ready for rendering.

### Claim 18 (depends on 15)

The apparatus of claim 15, wherein the processor is further configured to:
receive second input corresponding to a request to enable a normal display mode;
enable receipt of a third video stream, the third video stream having a lower resolution than the second resolution and a higher resolution than the first resolution; and
output the third video stream.

### Claim 19 (depends on 15)

The apparatus of claim 15, wherein the first video stream has a first frame rate and the second video stream has a second frame rate, wherein the second frame rate is a higher frame rate than the first frame rate.

### Claim 20 (depends on 15)

The apparatus of claim 15, wherein the region of interest is described by information including a bounding box and a mapping of the bounding box to the first frame.

### Claim 21 (depends on 15)

The apparatus of claim 15, further comprising:
a display for displaying the enlarged sections of the video frames.

### Claim 22 (depends on 15)

The apparatus of claim 15, wherein the apparatus comprises a mobile device with a camera.

### Claim 23 (depends on 15)

The apparatus of claim 15, wherein the request is for enabling an auto-zoom playing mode, wherein, in the auto-zoom playing mode, the enlarged sections of the video frames track the region of interest as the region of interest moves between the video frames.

### Claim 24 (depends on 15)

The apparatus of claim 15, wherein the processor is further configured to:
output the first video stream after the input is received and until the enlarged sections of the video frames are ready for rendering.

### Claim 25 (depends on 15)

The apparatus of claim 15, wherein the video frames of the second video stream are coded using tile-based video coding, wherein tile-based video coding divides each of the video frames into a plurality of tiles.

### Claim 26 (depends on 25)

The apparatus of claim 25, wherein each tile of the plurality of tiles is arranged in a separate track.

### Claim 27 (depends on 26)

The apparatus of claim 26, wherein the set of tiles are associated with a second video frame from the video frames, and wherein the processor is further configured to:
identify a second set of tiles in a third video frame from the video frames, the second set of tiles corresponding to the region of interest with respect to the third video frame, wherein the second set of tiles are different from the set of tiles, and wherein the enlarged sections of the video frames include the second set of tiles.

### Claim 28 (depends on 15)

The apparatus of claim 15, wherein the processor is further configured to
identify an object at the location;
identify a second region of interest in a second video frame, the second region of interest include an area around a second location of the object in the second video frame; and
identify a second set of tiles of the second video stream, the second set of tiles corresponding to the second region of interest, wherein the enlarged sections of the video frames include the second set of tiles.

### Claim 29

A non-transitory computer-readable medium having stored thereon instructions that when executed by a processor perform a method, comprising:
outputting a first video stream the first video stream having a first resolution;
receiving input corresponding to selection of a location in a first video frame of the first video stream, wherein the input is associated with a request to zoom in on the location in the first video frame;
determining a region of interest using the first video stream, the region of interest including an area around the location in the first video frame, wherein the region of interest is less than an entire area of the first video frame;
identifying a set of tiles of a second video stream, the set of tiles corresponding to the region of interest, the second video stream having a second resolution, wherein the second resolution is a higher resolution than the first resolution and wherein the set of tiles include fewer than all tiles within video frames of the second video stream;
enabling receipt of sections of the video frames of the second video stream, the sections of the video frames including the set of tiles and excluding areas of the video frames outside of the set of tiles;
enlarging the sections of the video frames according to a zoom factor; and
outputting the enlarged sections of the video frames.

### Claim 30 (depends on 29)

The non-transitory computer-readable medium of claim 29, wherein the first resolution is associated with a first bitrate, and wherein the first bitrate corresponds to an available bandwidth for receiving the first video stream.