System and method for very low frame rate video streaming for face-to-face video conferencing
AssigneeMicrosoft Corporation
InventorsMichael Cohen, Jue Wang
Life of this patent
Estimating the term…
Abstract
A real-time low frame-rate video compression system and method that allows the user to perform face-to-face communication through an extremely low bandwidth network. At the encoder side, the system is able to automatically select only a few good faces from the original sequence with high visual quality and compress and transmit them. At the decoder side, the system use image-morphing based rendering method to generate a normal frame-rate video. Experimental results show that the system is superior to more traditional video codecs for low bit-rate face-to-face communication.
Claims19
A process for encoding video data for face to face video conferencing comprising the process actions of:
- inputting a video frame of a video frame sequence some images of which contain a face;
- processing said video frame to locate said a face;
- if a face is found processing the face to locate features, but if no face is found no longer processing said frame;
- searching said face for features and using said found features to evaluate whether said frame is a good frame that should be encoded, wherein each good frame is selected from the input video frame sequence based on how frequently good frames are to be selected and both a face is found and the eyes of the face are open;
- if the frame is not a good frame, no longer processing that frame;
- subtracting said frame from said previously input frame to obtain a residual;
- and
- encoding said residual with a video encoder.
A process for decoding video data for face-to-face video conferencing, comprising the process actions of:
- receiving an encoded residual with control parameters based on features of a person's face;
- decoding said encoded residual and adding said decoded residual to a previously decoded frame to recover an image of a face;
- using said control parameters to unmorph the face in a new frame to its location in the previously decoded frame;
- putting the new frame in a buffer;
- and
- rendering a current display in real-time by morphing consecutive images in said buffer.
A video conferencing system for streaming face-to-face video of video conference participants, comprising:
- a general purpose computing device;
- and
- a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,
- input a video frame which possibly contains a face of a person participating in a video conference;
- process said video frame to locate a face box around said possible face;
- if a face box is found, process the face box to locate features, but if no face is found not process said frame any further;
- use said found features to evaluate whether said frame is a good frame that should be encoded based on whether the eyes are open and criteria regarding how frequently good frames are to be selected;
- if frame is not a good frame, no longer process that frame;
- if frame is a good frame, perform image morphing to align said frame with a previously input frame;
- subtract said frame from said previously input frame to obtain a residual;
- encode said residual with a video encoder;
- and
- transmit said encoded residual to other video participants.
Dependencies are inferred from the claim text.