# US 2020/0396501 A1 · Re-encoding predicted picture frames in live video stream applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0396501 A1
- Document ID: US20200396501A1
- Status: Published application
- Kind: A1
- Application number: 16/690,669
- Filed: 2019-11-21
- Published: 2020-12-17
- Earliest claimed: 2019-06-11
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Olivier Lapicque; Srinivas Anne
- Web page: https://patentgrep.com/patent/US20200396501A1

## CPC

- H04N 21/231: Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
- H04N 21/4402: involving reformatting operations of video signals for household redistribution, storage or real-time display {(details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764)}
- A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
- A63F 13/86: Watching games played by other players
- H04N 19/107: between spatial and temporal predictive coding, e.g. picture refresh
- H04N 19/124: Quantisation
- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/172: the region being a picture, frame or field
- H04N 19/177: the unit being a group of pictures [GOP]
- H04N 19/184: the unit being bits, e.g. of the compressed video stream
- H04N 19/40: using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/50: using predictive coding (H04N19/61 takes precedence)
- H04N 21/2187: Live feed
- H04N 21/2343: involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)}
- H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/433: Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N 21/434: Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream {(demultiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00)}
- H04N 21/4781: {Games}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N 21/854: Content authoring
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR

## Abstract

In various examples, a media stream may be received by a re-encode system that may leverage a recode engine to convert (e.g., at an interval, based on a request, etc.) an inter-frame associated with the media stream to an intra-frame. The intra-frame may be converted from the inter-frame using parameters or other information associated with and received with the media stream. The converted intra-frame may be merged into an updated segment of the media stream in place of the original inter-frame to enable storage of the updated segment—or a portion thereof—for later use.

## Claims

### Claim 1

A method comprising:
receiving, from a remotely located device, a bitstream representative of an encoded video sequence;
generating, using a decoder, display data for display based at least in part on a first instance of the bitstream;
determining, using a recode engine and based at least in part on a second instance of the bitstream, that an initial frame of a bitstream segment to be recorded is an inter-frame;
receiving, from the decoder, a decoded instance of the inter-frame, the decoded instance of the inter-frame being generated based at least in part on the first instance of the bitstream;
converting, using the recode engine and based at least in part on encoding parameters associated with the bitstream, the decoded instance of the inter-frame to generate an intra-frame; and
merging the intra-frame into the bitstream segment in place of the inter-frame; and
updating header information of one or more of the inter-frames subsequent the intra-frame such that the one or more inter-frames references the intra-frame.

### Claim 2 (depends on 1)

The method of claim 1, wherein the encoding parameters are determined from at least one of a Sequence Parameter Set (SPS) or a Picture Parameter Set (PPS) associated with the bitstream.

### Claim 3 (depends on 1)

The method of claim 1, wherein the encoding parameters include a Quantization Parameter (QP) used to establish a first quantizer value for the intra-frame, the QP being at least one step lower than a second quantizer value associated with the initial inter-frame.

### Claim 4 (depends on 1)

The method of claim 1, wherein the inter-frame is one of a Predicted (P) picture frame or a Bidirectional (B) picture frame, and the intra-frame is one of an Intra-coded (I) picture frame or an Instantaneous Decoder Refresh (IDR) frame.

### Claim 5 (depends on 1)

The method of claim 1, wherein the bitstream is an inter-frame only bitstream, and the updating the header information includes updating the header information for each of the inter-frames subsequent the intra-frame.

### Claim 6 (depends on 1)

The method of claim 1, further comprising determining to record the bitstream segment based at least in part on data representative of a request to capture the bitstream segment during the display of the first instance of the bitstream on a display of a client device, wherein the bitstream corresponds to a livestream and the bitstream segment corresponds to a portion of the livestream.

### Claim 7 (depends on 1)

The method of claim 1, wherein the second instance of the bitstream corresponds to a video format and the intra-frame is generated in the video format such that the updated bitstream segment corresponds to the video format.

### Claim 8 (depends on 1)

The method of claim 1, wherein the bitstream segment is generated in a first Motion Picture Experts Group (MPEG) format, and the method further comprises storing the updated bitstream segment in a second MPEG format different from the first MPEG format.

### Claim 9

A method comprising:
receiving, from a server and by a client device, first data representative of a sequence of inter-frames of a bitstream;
generating, using a decoder of the client device, display data based at least in part on a first instance of the sequence of inter-frames of the bitstream;
determining, at an interval, an inter-frame from the sequence of inter-frames of a second instance of the sequence of inter-frames of the bitstream;
receiving, based at least in part on the selecting, a decoded instance of the selected inter-frame from the decoder, the decoded instance of the selected inter-frame being generated based at least in part on the second instance of the sequence of inter-frames of the bitstream;
converting the decoded instance of the selected inter-frame to an intra-frame based at least in part on encoding parameters associated with the first data; and
merging second data representative of the intra-frame into the second instance of the sequence of inter-frames of the bitstream in place of the inter-frame to generate an updated instance of the bitstream.

### Claim 10 (depends on 9)

The method of claim 9, further comprising capturing a segment of the updated instance of the bitstream, the segment beginning with the intra-frame.

### Claim 11 (depends on 9)

The method of claim 9, wherein the encoding parameters are determined from at least one of a Sequence Parameter Set (SPS) or a Picture Parameter Set (PPS) associated with the bitstream.

### Claim 12 (depends on 9)

The method of claim 9, wherein the generating the updated instance of the bitstream includes updating a header of each inter-frame after the intra-frame and prior to a next intra-frame to reference the intra-frame.

### Claim 13 (depends on 9)

The method of claim 9, wherein the bitstream is a livestream of an instance of a game received by the client device within a cloud gaming environment, the bitstream representing the instance of the game as rendered by the server remote from the client device.

### Claim 14 (depends on 9)

The method of claim 9, further comprising:
receiving data representative of another request to generate a bitstream segment during the display of the first instance of the sequence of inter-frames of the bitstream on a display of the client device;
determining that the intra-frame is a closest intra-frame to a requested start point of bitstream segment; and
generating the bitstream segment from the updated instance of the bitstream, the bitstream segment starting with the intra-frame.

### Claim 15

A system comprising:
a computing device including one or more processing devices and one or more memory devices communicatively coupled to the one or more processing devices storing programmed instructions thereon, which when executed by the one or more processing devices cause the instantiation of:
a bitstream provider to receive first data representative of a bitstream;
a decoder to generate display data for display based at least in part on a first instance of the bitstream;
a frame selector to determine that at least an initial portion of a segment of the bitstream to be recorded includes only inter-frames;
an intra-frame encoder to:
receive, from the decoder, a decoded instance of an initial inter-frame of the initial portion of the segment; and
convert, based at least in part on encoding parameters associated with the bitstream, a decoded instance of the initial inter-frame to an intra-frame;
a frame merger to generate an updated segment based at least in part on:
merging the intra-frame into the segment of the second instance of the bitstream in place of the initial inter-frame; and
updating headers of one or more of the inter-frames subsequent the intra-frame.

### Claim 16 (depends on 15)

The system of claim 15, wherein the encoding parameters are determined from at least one of a Sequence Parameter Set (SPS) or a Picture Parameter Set (PPS) associated with the bitstream.

### Claim 17 (depends on 15)

The system of claim 15, wherein the encoding parameters include a Quantization Parameter (QP) used to establish first quantizer value for the intra-frame, the QP being at least one step lower than a second quantizer value associated with the initial inter-frame.

### Claim 18 (depends on 15)

The system of claim 15, wherein the bitstream corresponds to an inter-frame only group of pictures (GOP) structure, and the updating the headers includes updating the headers of each inter-frame subsequent the intra-frame.

### Claim 19 (depends on 15)

The system of claim 15, wherein the frame selector is further to request the decoded instance responsive to a request to generate the segment, the request received based at least in part on an input to an input device of the client device.

### Claim 20 (depends on 15)

The system of claim 15, wherein the bitstream is received in a first Motion Picture Experts Group (MPEG) format, and the updated segment is stored in a second MPEG format different from the first MPEG format.
