Temporal and spatial scaleable coding for video object planes

A method for scaling an input video sequence comprising video object planes (VOPs) for communication in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, comprising the steps of:

• providing a first particular one of said VOPs of said input video sequence for communication in said base layer as a first base layer VOP;
• downsampling pixel data of at least a portion of said first base layer VOP for communication in said enhancement layer as a first downsampled VOP at a temporal position corresponding to said first base layer VOP;
• downsampling corresponding pixel data of said first particular one of said VOPs to provide a comparison VOP;
• differentially encoding said first downsampled VOP using said comparison VOP;
• differentially encoding said first base layer VOP using said first particular one of said VOPs by:
• determining a residue according to a difference between pixel data of said first base layer VOP and pixel data of said first particular one of said VOPs;
• and
• spatially transforming said residue to provide transform coefficients;
• wherein said VOPs in said input video sequence are field mode VOPs, and said first base layer VOP is differentially encoded by reordering lines of said pixel data of said first base layer VOP in a field mode prior to said determining step if said lines of pixel data meet a reordering criteria.

A method for coding a bi-directionally predicted video object plane (B-VOP), comprising the steps of:

• scaling an input video sequence comprising video object planes (VOPs) for communication in a corresponding base layer and enhancement layer;
• providing first and second base layer VOPs in said base layer which correspond to said input video sequence VOPs;
• said second base layer VOP being predicted from said first base layer VOP according to a motion vector MV.sub.p ;
• providing said B-VOP in said enhancement layer at a temporal position which is intermediate to that of said first and second base layer VOPs;
• and
• encoding said B-VOP using at least one of:
• (a) a forward motion vector MV.sub.f and
• (b) a backward motion vector MV.sub.B, obtained by scaling said motion vector MV.sub.p.

A method for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• pixel data of a first particular one of said VOPs of said input video sequence is downsampled and carried as a first base layer VOP having a reduced spatial resolution;
• pixel data of at least a portion of said first base layer VOP is upsampled and carried as a first upsampled VOP in said enhancement layer at a temporal position corresponding to said first base layer VOP;
• and
• said first upsampled VOP is differentially encoded using said first particular one of said VOPs of said input video sequence;
• said method comprising the steps of:
• upsampling said pixel data of said first base layer VOP to restore said associated spatial resolution;
• and
• processing said first upsampled VOP and said first base layer VOP with said restored associated spatial resolution to provide an output video signal with said associated spatial resolution;
• wherein:
• a second particular one of said VOPs of said input video sequence is downsampled to provide a second base layer VOP having a reduced spatial resolution;
• pixel data of at least a portion of said second base layer VOP is upsampled to provide a second upsampled VOP in said enhancement layer which corresponds to said first upsampled VOP;
• at least one of said first and second base layer VOPs is used to predict an intermediate VOP corresponding to said first and second upsampled VOPs;
• and
• said intermediate VOP is encoded for communication in said enhancement layer at a temporal position which is intermediate to that of said first and second upsampled VOPs.

A method for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• a first particular one of said VOPs of said input video sequence is provided in said base layer as a first base layer VOP;
• pixel data of at least a portion of said first base layer VOP is downsampled and carried in said enhancement layer as a first downsampled VOP at a temporal position corresponding to said first base layer VOP;
• corresponding pixel data of said first particular one of said VOPs is downsampled to provide a comparison VOP;
• and
• said first downsampled VOP is differentially encoded using said comparison VOP;
• said method comprising the steps of:
• upsampling said pixel data of said first downsampled VOP to restore said associated spatial resolution;
• and
• processing said first enhancement layer VOP with said restored associated spatial resolution and said first base layer VOP to provide an output video signal with said associated spatial resolution;
• wherein:
• said first base layer VOP is differentially encoded using said first particular one of said VOPs by determining a residue according to a difference between pixel data of said first base layer VOP and pixel data of said first particular one of said VOPs, and spatially transforming said residue to provide transform coefficients;
• and
• said VOPs in said input video sequence are field mode VOPs, and said first base layer VOP is differentially encoded by reordering lines of said pixel data of said first base layer VOP in a field mode prior to determining said residue if said lines of pixel data meet a reordering criteria.

A method for recovering an input video sequence comprising video object planes (VOPs) which was scaled and communicated in a corresponding base layer and enhancement layer in a data stream, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• first and second base layer VOPs are provided in said base layer which correspond to said input video sequence VOPs;
• said second base layer VOP is predicted from said first base layer VOP according to a motion vector MV.sub.B ;
• a bi-directionally predicted video object plane (B-VOP) is provided in said enhancement layer at a temporal position which is intermediate to that of said first and second base layer VOPs;
• and
• said B-VOP is encoded using a forward motion vector MV.sub.f and a backward motion vector MV.sub.p which are obtained by scaling said motion vector MV.sub.p ;
• said method comprising the steps of:
• recovering said forward motion vector MV.sub.f and said backward motion vector MV.sub.B from said data stream;
• and
• decoding said B-VOP using said forward motion vector MV.sub.f and said backward motion vector MV.sub.B.

A decoder apparatus for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• pixel data of a first particular one of said VOPs of said input video sequence is downsampled and carried as a first base layer VOP having a reduced spatial resolution;
• pixel data of at least a portion of said first base layer VOP is upsampled and carried as a first upsampled VOP in said enhancement layer at a temporal position corresponding to said first base layer VOP;
• and
• said first upsampled VOP is differentially encoded using said first particular one of said VOPs of said input video sequence;
• said apparatus comprising:
• means for upsampling said pixel data of said first base layer VOP to restore said associated spatial resolution;
• and
• means for processing said first upsampled VOP and said first base layer VOP with said restored associated spatial resolution to provide an output video signal with said associated spatial resolution;
• wherein:
• said VOPs in said input video sequence are field mode VOPs;
• and
• said first upsampled VOP is differentially encoded by reordering lines of said pixel data of said first upsampled VOP in a field mode if said lines of pixel data meet a reordering criteria, then determining a residue according to a difference between pixel data of said first unsampled VOP and pixel data of said first particular one of said VOPs of said input video sequence, and spatially transforming said residue to provide transform coefficients.

A decoder apparatus for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• a first particular one of said VOPs of said input video sequence is provided in said base layer as a first base layer VOP;
• pixel data of at least a portion of said first base layer VOP is downsampled and carried in said enhancement layer as a first downsampled VOP at a temporal position corresponding to said first base layer VOP;
• corresponding pixel data of said first particular one of said VOPs is downsampled to provide a comparison VOP;
• and
• said first downsampled VOP is differentially encoded using said comparison VOP;
• said apparatus comprising:
• means for upsampling said pixel data of said first downsampled VOP to restore said associated spatial resolution;
• and
• means for processing said first enhancement layer VOP with said restored spatial resolution and said first base layer VOP to provide an output video signal with said associated spatial resolution;
• wherein:
• said first downsampled VOP is differentially encoded by determining a residue according to a difference between pixel data of said first downsampled VOP and pixel data of said first particular one of said VOPs of said input video sequence, and spatially transforming said residue to provide transform coefficients;
• and
• said VOPs in said input video sequence are field mode VOPs, and said first base layer VOP is differentially encoded by reordering lines of said pixel data of said first base layer VOP in a field mode prior to determining said residue if said lines of pixel data meet a reordering criteria.

A decoder apparatus for recovering an input video sequence comprising video object planes (VOPs) which was scaled and communicated in a corresponding base layer and enhancement layer in a data stream, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• first and second base layer VOPs which correspond to said input video sequence VOPs are provided in said base layer;
• said second base layer VOP is predicted from said first base layer VOP according to a motion vector MV.sub.p ;
• a bi-directionally predicted video object plane (B-VOP) is provided in said enhancement layer at a temporal position which is intermediate to that of said first and second base layer VOPs;
• and
• said B-VOP is encoded using a forward motion vector MV.sub.f and a backward motion vector MV.sub.B which are obtained by scaling said motion vector MV.sub.p ;
• said apparatus comprising:
• means for recovering said forward motion vector MV.sub.f and said backward motion vector MV.sub.B from said data stream;
• and
• means for decoding said B-VOP using said forward motion vector MV.sub.f and said backward motion vector MV.sub.B.

A method for scaling an input video sequence comprising video object planes (VOPs) for communication in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, comprising the steps of:

• downsampling pixel data of a first particular one of said VOPs of said input video sequence to provide a first base layer VOP having a reduced spatial resolution;
• upsampling pixel data of at least a portion of said first base layer VOP to provide a first upsampled VOP in said enhancement layer;
• differentially encoding said first upsampled VOP using said first particular one of said VOPs of said input video sequence for communication in said enhancement layer at a temporal position corresponding to said first base layer VOP;
• wherein said VOPs in said input video sequence are field mode VOPs, and said differentially encoding step comprises the further steps of:
• reordering lines of said pixel data of said first upsampled VOP in a field mode if said lines of pixel data meet a reordering criteria;
• then
• determining a residue according to a difference between pixel data of said first upsampled VOP and pixel data of said first particular one of said VOPs of said input video sequence;
• and
• spatially transforming said residue to provide transform coefficients.

A method for scaling an input video sequence comprising video object planes (VOPs) for communication in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, comprising the steps of:

• downsampling pixel data of a first particular one of said VOPs of said input video sequence to provide a first base layer VOP having a reduced spatial resolution;
• upsampling pixel data of at least a portion of said first base layer VOP to provide a first upsampled VOP in said enhancement layer;
• and
• differentially encoding said first upsampled VOP using said first particular one of said VOPs of said input video sequence for communication in said enhancement layer at a temporal position corresponding to said first base layer VOP;
• wherein:
• said base layer is adapted to carry higher priority, lower bit rate data, and said enhancement layer is adapted to carry lower priority, higher bit rate data.

A method for scaling an input video sequence comprising video object planes (VOPs) for communication in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, comprising the steps of:

• providing a first particular one of said VOPs of said input video sequence for communication in said base layer as a first base layer VOP;
• downsampling pixel data of at least a portion of said first base layer VOP for communication in said enhancement layer as a first downsampled VOP at a temporal position corresponding to said first base layer VOP;
• downsampling corresponding pixel data of said first particular one of said VOPs to provide a comparison VOP;
• differentially encoding said first downsampled VOP using said comparison VOP;
• providing a second particular one of said VOPs of said input video sequence for communication in said base layer as a second base layer VOP;
• downsampling pixel data of at least a portion of said second base layer VOP for communication in said enhancement layer as a second downsampled VOP at a temporal position corresponding to said second base layer VOP;
• downsampling corresponding pixel data of said second particular one of said VOPs to provide a comparison VOP;
• differentially encoding said second downsampled VOP using said comparison VOP;
• using at least one of said first and second base layer VOPs to predict an intermediate VOP corresponding to said first and second downsampled VOPs;
• and
• encoding said intermediate VOP for communication in said enhancement layer at a temporal position which is intermediate to that of said first and second downsampled VOPs.

A method for scaling an input video sequence comprising video object planes (VOPs) for communication in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, comprising the steps of:

• providing a first particular one of said VOPs of said input video sequence for communication in said base layer as a first base layer VOP;
• downsampling pixel data of at least a portion of said first base layer VOP for communication in said enhancement layer as a first downsampled VOP at a temporal position corresponding to said first base layer VOP;
• downsampling corresponding pixel data of said first particular one of said VOPs to provide a comparison VOP;
• and
• differentially encoding said first downsampled VOP using said comparison VOP;
• wherein:
• the base and enhancement layers are adapted to provide a stereoscopic video capability in which image data in the enhancement layer has a lower spatial resolution than image data in the base layer.

A method for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• pixel data of a first particular one of said VOPs of said input video sequence is downsampled and carried as a first base layer VOP having a reduced spatial resolution;
• pixel data of at least a portion of said first base layer VOP is upsampled and carried as a first upsampled VOP in said enhancement layer at a temporal position corresponding to said first base layer VOP;
• and
• said first upsampled VOP is differentially encoded using said first particular one of said VOPs of said input video sequence;
• said method comprising the steps of:
• upsampling said pixel data of said first base layer VOP to restore said associated spatial resolution;
• and
• processing said first upsampled VOP and said first base layer VOP with said restored associated spatial resolution to provide an output video signal with said associated spatial resolution;
• wherein:
• said VOPs in said input video sequence are field mode VOPs;
• and
• said first upsampled VOP is differentially encoded by reordering lines of said pixel data of said first upsampled VOP in a field mode if said lines of pixel data meet a reordering criteria, then determining a residue according to a difference between pixel data of said first upsampled VOP and pixel data of said first particular one of said VOPs of said input video sequence, and spatially transforming said residue to provide transform coefficients.

A method for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• pixel data of a first particular one of said VOPs of said input video sequence is downsampled and carried as a first base layer VOP having a reduced spatial resolution;
• pixel data of at least a portion of said first base layer VOP is upsampled and carried as a first upsampled VOP in said enhancement layer at a temporal position corresponding to said first base layer VOP;
• and
• said first upsampled VOP is differentially encoded using said first particular one of said VOPs of said input video sequence;
• said method comprising the steps of:
• upsampling said pixel data of said first base layer VOP to restore said associated spatial resolution;
• and
• processing said first upsampled VOP and said first base layer VOP with said restored associated spatial resolution to provide an output video signal with said associated spatial resolution;
• wherein:
• said base layer is adapted to carry higher priority, lower bit rate data, and said enhancement layer is adapted to carry lower priority, higher bit rate data.

A method for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• a first particular one of said VOPs of said input video sequence is provided in said base layer as a first base layer VOP;
• pixel data of at least a portion of said first base layer VOP is downsampled and carried in said enhancement layer as a first downsampled VOP at a temporal position corresponding to said first base layer VOP;
• corresponding pixel data of said first particular one of said VOPs is downsampled to provide a comparison VOP;
• and
• said first downsampled VOP is differentially encoded using said comparison VOP;
• said method comprising the steps of:
• upsampling said pixel data of said first downsampled VOP to restore said associated spatial resolution;
• and
• processing said first enhancement layer VOP with said restored associated spatial resolution and said first base layer VOP to provide an output video signal with said associated spatial resolution;
• wherein:
• a second particular one of said VOPs of said input video sequence is provided in said base layer as a second base layer VOP;
• pixel data of at least a portion of said second base layer VOP is downsampled and carried in said enhancement layer as a second downsampled VOP at a temporal position corresponding to said second base layer VOP;
• corresponding pixel data of said second particular one of said VOPs is downsampled to provide a comparison VOP;
• said second downsampled VOP is differentially encoded using said comparison VOP;
• at least one of said first and second base layer VOPs is used to predict an intermediate VOP corresponding to said first and second downsampled VOPs;
• and
• said intermediate VOP is encoded for communication in said enhancement layer at a temporal position which is intermediate to that of said first and second downsampled VOPs.

A method for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• a first particular one of said VOPs of said input video sequence is provided in said base layer as a first base layer VOP;
• pixel data of at least a portion of said first base layer VOP is downsampled and carried in said enhancement layer as a first downsampled VOP at a temporal position corresponding to said first base layer VOP;
• corresponding pixel data of said first particular one of said VOPs is downsampled to provide a comparison VOP;
• and
• said first downsampled VOP is differentially encoded using said comparison VOP;
• said method comprising the steps of:
• upsampling said pixel data of said first downsampled VOP to restore said associated spatial resolution;
• and
• processing said first enhancement layer VOP with said restored associated spatial resolution and said first base layer VOP to provide an output video signal with said associated spatial resolution;
• wherein:
• said base and enhancement layer are adapted to provide a stereoscopic video capability in which image data in said enhancement layer has a lower spatial resolution than image data in said base layer.

A decoder apparatus for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• pixel data of a first particular one of said VOPs of said input video sequence is downsampled and carried as a first base layer VOP having a reduced spatial resolution;
• pixel data of at least a portion of said first base layer VOP is upsampled and carried as a first upsampled VOP in said enhancement layer at a temporal position corresponding to said first base layer VOP;
• and
• said first upsampled VOP is differentially encoded using said first particular one of said VOPs of said input video sequence;
• said apparatus comprising:
• means for upsampling said pixel data of said first base layer VOP to restore said associated spatial resolution;
• and
• means for processing said first upsampled VOP and said first base layer VOP with said restored associated spatial resolution to provide an output video signal with said associated spatial resolution;
• wherein:
• a second particular one of said VOPs of said input video sequence is downsampled to provide a second base layer VOP having a reduced spatial resolution;
• pixel data of at least a portion of said second base layer VOP is upsampled to provide a second upsampled VOP in said enhancement layer which corresponds to said first upsampled VOP;
• at least one of said first and second base layer VOPs is used to predict an intermediate VOP corresponding to said first and second upsampled VOPs;
• and
• said intermediate VOP is encoded for communication in said enhancement layer at a temporal position which is intermediate to that of said first and second upsampled VOPs.

A decoder apparatus for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• pixel data of a first particular one of said VOPs of said input video sequence is downsampled and carried as a first base layer VOP having a reduced spatial resolution;
• pixel data of at least a portion of said first base layer VOP is upsampled and carried as a first upsampled VOP in said enhancement layer at a temporal position corresponding to said first base layer VOP;
• and
• said first upsampled VOP is differentially encoded using said first particular one of said VOPs of said input video sequence;
• said apparatus comprising:
• means for upsampling said pixel data of said first base layer VOP to restore said associated spatial resolution;
• and
• means for processing said first upsampled VOP and said first base layer VOP with said restored associated spatial resolution to provide an output video signal with said associated spatial resolution;
• wherein:
• said base layer is adapted to carry higher priority, lower bit rate data, and said enhancement layer is adapted to carry lower priority, higher bit rate data.

A decoder apparatus for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• a first particular one of said VOPs of said input video sequence is provided in said base layer as a first base layer VOP;
• pixel data of at least a portion of said first base layer VOP is downsampled and carried in said enhancement layer as a first downsampled VOP at a temporal position corresponding to said first base layer VOP;
• corresponding pixel data of said first particular one of said VOPs is downsampled to provide a comparison VOP;
• and
• said first downsampled VOP is differentially encoded using said comparison VOP;
• said apparatus comprising:
• means for upsampling said pixel data of said first downsampled VOP to restore said associated spatial resolution;
• and
• means for processing said first enhancement layer VOP with said restored spatial resolution and said first base layer VOP to provide an output video signal with said associated spatial resolution;
• wherein:
• a second particular one of said VOPs of said input video sequence is provided for communication in said base layer as a second base layer VOP;
• pixel data of at least a portion of said second base layer VOP is downsampled to provide a second downsampled VOP in said enhancement layer which corresponds to said first upsampled VOP;
• at least one of said first and second base layer VOPs is used to predict an intermediate VOP corresponding to said first and second downsampled VOPs;
• and
• said intermediate VOP is encoded for communication in said enhancement layer at a temporal position which is intermediate to that of said first and second downsampled VOPs.

A decoder apparatus for recovering an input video sequence comprising video object planes (VOPs) which were scaled and communicated in a corresponding base layer and enhancement layer, said VOPs in said input video sequence having an associated spatial resolution and temporal resolution, wherein:

• a first particular one of said VOPs of said input video sequence is provided in said base layer as a first base layer VOP;
• pixel data of at least a portion of said first base layer VOP is downsampled and carried in said enhancement layer as a first downsampled VOP at a temporal position corresponding to said first base layer VOP;
• corresponding pixel data of said first particular one of said VOPs is downsampled to provide a comparison VOP;
• and
• said first downsampled VOP is differentially encoded using said comparison VOP;
• said apparatus comprising:
• means for upsampling said pixel data of said first downsampled VOP to restore said associated spatial resolution;
• and
• means for processing said first enhancement layer VOP with said restored spatial resolution and said first base layer VOP to provide an output video signal with said associated spatial resolution;
• wherein:
• said base and enhancement layer are adapted to provide a stereoscopic video capability in which image data in said enhancement layer has a lower spatial resolution than image data in said base layer.