US 10,313,683 B2Granted

Video encoder with context switching

AssigneeApple Inc.

InventorsJoseph J. Cheng, Timothy J. Millet, Shun Wai Go, Guy Cote

Life of this patent

Estimating the term…

Loading patent…

Abstract

A context switching method for video encoders that enables higher priority video streams to interrupt lower priority video streams. A high priority frame may be received for processing while another frame is being processed. The pipeline may be signaled to perform a context stop for the current frame. The pipeline stops processing the current frame at an appropriate place, and propagates the stop through the stages of the pipeline and to a transcoder through DMA. The stopping location is recorded. The video encoder may then process the higher-priority frame. When done, a context restart is performed and the pipeline resumes processing the lower-priority frame beginning at the recorded location. The transcoder may process data for the interrupted frame while the higher-priority frame is being processed in the pipeline, and similarly the pipeline may begin processing the lower-priority frame after the context restart while the transcoder completes processing the higher-priority frame.

Claims16

Dependent claims

An apparatus, comprising:

a block processing pipeline comprising a plurality of stages each configured to perform one or more operations on blocks of pixels from input frames passing through the pipeline, wherein at least one of the plurality of stages is configured to access pixel data from one or more previously processed neighbor blocks on a row above a current block being processed to perform an operation on a rent block of pixels being processed at the stage;
wherein the block processing pipeline is configured to:
begin processing a frame in a first video stream from a first video source, wherein, to process the frame, blocks of pixels are read from the frame by a first stage of the pipeline and passed through the plurality of stages of the pipeline;
receive, at the first stage of the block processing pipeline, an indication that a higher priority frame in a second video stream from a second video source needs to be processed;
responsive to receipt of the indication, continue processing at least one pixel of the frame in the first video stream from the first video source until the first stage determines that a context stop location within the frame has been reached, wherein the context stop location is reached prior to completion of processing of the frame in the first video stream;
perform a context stop for the frame from the first video source at the context stop location within the frame, wherein, in performing the context stop, a last row of processed blocks of the frame is stored to a buffer and an indication of the context stop location is propagated from the first stage through one or more downstream stages of the pipeline to notify the downstream stages that a context stop is being performed at the context stop location in the frame;
process the higher priority frame from the second video source;
and
perform a context restart to resume processing of the frame from the first video source at within the frame, wherein, in performing the context restart, the last row of processed blocks of the frame is read from the buffer.

6 more elements

A method, comprising:

inputting blocks of pixels from a current frame to a block processing pipeline comprising a plurality of stages, each stage configured to perform one or more operations on each block of pixels, wherein at least one stage is configured to access a previous row of processed blocks in performing operations on blocks in a current row of blocks being processed;
receiving an indication of another frame to be processed in the block processing pipeline, wherein the other frame is a higher priority frame from a different video source than the current frame;
in response to the indication, performing a context stop comprising:
directing the block processing pipeline to stop processing the current frame at a context stop location after a current processing location and prior to an end of the current frame;
determining that the block processing pipeline has stopped processing the current frame;
storing a last row of processed blocks of the current frame to a buffer in response to the determination;
in response to completion of the context stop, processing the other frame in the block processing pipeline;
in response to completion of the processing, performing a context restart comprising:
restoring the last row of processed blocks of the current frame from the buffer;
and
directing the block processing pipeline to resume processing of the current frame at the context stop location within the frame, wherein at least one stage accesses the restored last row of processed blocks in performing operations on a first row of input blocks after resuming processing of the current frame at the context stop location.

8 more elements

A device, comprising:

a memory;
and
an apparatus comprising a block processing pipeline that includes a plurality of stages configured to process blocks of pixels passing through the pipeline and output the processed blocks to the memory, wherein at least one stage is configured to access a previous row of processed blocks in performing operations on blocks in a current row of blocks being processed, wherein the apparatus is configured to:
begin processing a current frame in the block processing pipeline;
receive an indication that another frame needs to be processed in the block processing pipeline;
direct the block processing pipeline to stop processing the current frame at a context stop location after a current processing location and prior to an end of the current frame;
store, to perform a context stop, a last row of processed blocks of the current frame to a buffer in response to a determination that the block processing pipeline has stopped processing the current frame;
in response to completion of the context stop, process the other frame in the block processing pipeline;
restore, to perform a context restore, the last row of processed blocks of the current frame from the buffer;
and
direct the block processing pipeline to resume processing of the current frame at the context stop location within the frame, wherein at least one stage accesses the restored last row of processed blocks in performing operations on a first row of input blocks after resuming processing of the current frame at the context stop location;
wherein the current frame is from a first video source, wherein the other frame is from a higher priority video source than the first video source.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 14/474,114Filed 30 Aug 2014
Published: US 2016/0065969 A13 Mar 2016
Granted: 4 Jun 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification13 codes

H04N 19/176
The unit being an image region, e.g. an object: the region being a block, e.g. a macroblock
G06F 9/461
Multiprogramming arrangements: saving or restoring of program or task context
G06F 9/4818
By interrupt, e.g. masked: priority circuits therefor
H04N 19/127
Characterised by the element, parameter or selection affected or controlled by the adaptive coding: prioritisation of hardware or computational…

Family

US 2016/0065969 A1Pre-grant publication · published 3 Mar 2016
US 10,313,683 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key