Method and system for real-time video communication within a virtual environment

A method of communication comprising:

• a) creating a virtual environment in three dimensions;
• b) capturing a plurality of real-time video streams of a local participant from a plurality of sample viewpoints;
• c) generating a plurality of output video image streams in real-time of said local participant, to be displayed to a plurality of observing participants, by applying a new view synthesis technique to said plurality of real-time video streams, said plurality of output video image streams rendered from a plurality of observing locations of said plurality of observing participants with respect to a first location of said local participant within a coordinate space of said virtual environment;
• d) enabling changes in at least one of said first location and said plurality of observing locations;
• e) modifying corresponding output video image streams in said plurality of output video image streams to reflect said changes in c);
• and
• f) enabling real-time video communication in a communication session from said local participant to said plurality of observing participants.

A video conferencing station comprising:

• a plurality of video cameras for recording real-time video streams of a local participant at multiple sample viewpoints surrounding said user;
• a new view synthesis module for generating a plurality of output video image streams by applying a new view synthesis technique to said real-time video streams from said plurality of video cameras of said local participant rendered from a plurality of locations of a plurality of observing participants with respect to a first location of said local participant within a coordinate space of a virtual environment;
• an interface module for blending a plurality of input video image streams from said plurality of observing participants within a synthetic rendering of said three dimensional virtual environment;
• and
• a display for viewing said plurality of input video image streams to enable real-time video communication in a communication session between said local participant and said plurality of observing participants within said virtual environment.

A method of communication comprising:

• a) simultaneously generating a first video image stream in real-time of a first participant and a second video image stream in real-time of a second participant by applying a new view synthesis technique to a first plurality of real-time video streams of said first participant and to a second plurality of real-time video streams of said second participant said first and second participants represented within a virtual environment,
• b) allowing for navigation within said virtual environment by said first and second participants;
• c) modifying said first and second video image streams to reflect said navigation in b)
• d) sending said first video image stream to said second participant;
• e) sending said second video image stream to said first participant;
• and
• f) enabling real-time video communication between said first and second participant within said virtual environment.

A computer system comprising:

• a processor:
• and
• a computer readable medium coupled to said processor and containing program instructions that, when executed, cause said processor to implement a method of video conferencing on said computer system comprising:
• a) creating a virtual environment in three dimensions;
• b) capturing a plurality of real-time video streams of a local participant from a plurality of sample viewpoints;
• c) generating a video image stream in real-time of said local participant to be displayed to a second participant by applying a new view synthesis technique to said plurality of real-time video streams, said video image stream rendered from a second location of said second participant with respect to a first location of said local participant within a coordinate space of said virtual environment;
• d) enabling a change in at least one of said first and second locations;
• e) modifying said video image stream to reflect said change in c);
• and
• f) enabling real-time video communication from said local participant to said second participant.