Low delay real time digital video mixing for multipoint video conferencing

A method of operating a multipoint control unit for use in videoconferencing of a plurality of audio-video terminals comprises:

• feeding a plurality of video streams to the multipoint control unit;
• for each one of said streams, extracting segments of data and adaptively reducing the bit rates;
• outputting the segments of data to a data queue;
• forming new picture data units from the segments of data stored in each of the data queues by:
• including the new picture data from those queues that have a complete picture;
• and if no queue has a complete picture,
• determining if for each of the queues whether the queues have a complete segment of data and for those queues having a segment of data including data from those queues in the new picture.

A multipoint control unit comprises:

• a plurality of segment processors arranged to extract segment data from a corresponding plurality of input video streams;
• a plurality of data queues, each one of said data queues being coupled to a corresponding one of said plurality of segment processors for storing segment data provided from said segment processors;
• a segment combiner coupled to said data queues for combining output data from said data queues;
• and
• a controller coupled to said segment combiner for selectively providing rate control data to said segment processors in accordance with the amount of the data in said data queues.

A method of operating a multipoint control unit for use in a multipoint video conference comprises:

• extracting segment data from a plurality of video streams;
• storing the segment data in a one of a plurality of data queues;
• determining whether there is data in any of the plurality of data queues that can be included in a new picture;
• and
• combining data from said plurality of data queues in accordance with a determination of whether there is data in the data queues;
• wherein the determining further comprises:
• determining a number of segments of data awaiting service for each of the queues.

A method of operating a multipoint control unit for use in a multipoint video conference comprises:

• extracting segment data from a plurality of video streams;
• storing the segment data in a one of a plurality of data queues;
• determining whether there is data in any of the plurality of data queues that can be included in a new picture;
• and
• combining data from said plurality of data queues in accordance with a determination of whether there is data in the data queues;
• wherein said extracting further comprises:
• selectively reducing output bit rates from each one of said data queues;
• wherein said combining further comprises:
• combining data from said data queues to provide an overall output video bit stream syntax from the multipoint control unit which is in accordance with a syntax expected from a remote terminal.

A computer program product residing on a computer readable medium comprising instructions for causing a multipoint control unit to:

• extract segment data to provide a plurality of extracted segment streams;
• store each one of the segment data streams in a one of a plurality of data queues;
• determine which of said data queues has data for inclusion in a new picture;
• and
• combine data from said plurality of data queues in accordance with a determination of which data queues has data for inclusion in the new picture;
• wherein an output picture rate of the combined video bit stream is related to the maximum input picture rate of said input video streams;
• wherein instructions that causes the multipoint control unit to determine if there is data in the queues further comprises instructions to:
• determine if any of the queues has a complete picture of data awaiting transmission.

A computer program product residing on a computer readable medium comprising instructions for causing a multipoint control unit to:

• extract segment data to provide a plurality of extracted segment streams;
• store each one of the segment data streams in a one of a plurality of data queues;
• determine which of said data queues has data for inclusion in a new picture;
• and
• combine data from said plurality of data queues in accordance with a determination of which data queues has data for inclusion in the new picture;
• wherein an output picture rate of the combined video bit stream is related to the maximum input picture rate of said input video streams;
• wherein instructions that causes the multipoint control unit to extract further comprises instructions to:
• reduce output bit rates from the extracted segment data streams;
• and
• combine data from said data queues to provide an overall output video bit stream from the multipoint control unit which is in accordance with a syntax expected by a remote terminal.

A computer program product residing on a computer readable medium comprising instructions for causing a multipoint control unit to:

• extract segment data to provide a plurality of extracted segment streams;
• store each one of the segment data streams in a one of a plurality of data queues;
• determine which of said data queues has data for inclusion in a new picture;
• and
• combine data from said plurality of data queues in accordance with a determination of which data queues has data for inclusion in the new picture;
• wherein an output picture rate of the combined video bit stream is related to the maximum input picture rate of said input video streams;
• wherein said bit rate control is derived from the queue fullness of said data queues.