View rendering for the provision of virtual eye contact using special geometric constraints in combination with eye-tracking

A video communication system comprising:

• an interface for receiving an inbound image or a sequence of inbound images;
• an inbound image analyzer for analyzing the inbound image or the sequence of inbound images and configured to estimate a location of a sought image feature within the inbound image or the sequence of inbound images;
• a display position determiner for determining a display position of the sought image feature when the inbound image or the sequence of inbound images is displayed on a display for a local participant of a video communication;
• a local participant image interface for receiving a local participant image or a sequence of local participant images depicting at least a part of the local participant;
• a local participant position determiner for determining a position of the local participant on the basis of the local participant image or the sequence of local participant images;
• a virtual camera pose determiner for providing a virtual camera pose on basis of the display position and the position of the local participant;
• a virtual view renderer for rendering a virtual view showing at least the local participant on the basis of the local participant image or the sequence of local participant images and the virtual camera pose, so that the virtual view shows the local participant from a perspective according to the virtual camera pose;
• at least one further local participant image interface for receiving a further local participant image or a sequence of further local participant images showing the local participant from a different perspective than the local participant image or the sequence of local participant images;
• and
• a 3D reconstruction component that analyzes the local participant image or the sequence of local participant images and the further local participant image or the sequence of further local participant images with respect to 3D information regarding the local participant.

A method for determining a position and an orientation of a virtual camera, the method comprising:

• receiving a displayed representation of a remote participant as displayed by a display;
• capturing a view of a local participant located in an environment of the display;
• determining a display position of the displayed representation of the remote participant;
• determining a position of a local participant on the basis of the view of the local participant;
• determining the position the virtual camera on the basis of the display position;
• determining the orientation of the virtual camera on the basis of the display position and the position of the local participant;
• rendering a virtual view of the local participant as if viewed by the virtual camera;
• receiving a further local participant image or a sequence of further local participant images showing the local participant from a different perspective than the view of the local participant;
• and
• analyzing the view of the local participant and the further local participant image or the sequence of further local participant images with respect to 3D information regarding the local participant.

A non-transitory computer readable digital storage medium having stored thereon a computer program comprising a program code for performing, when running on a computer, a method for determining a position and an orientation of a virtual camera, the method comprising:

• receiving a displayed representation of a remote participant as displayed by a display;
• capturing a view of a local participant located in an environment of the display;
• determining a display position of the displayed representation of the remote participant;
• determining a position of a local participant on the basis of the view of the local participant;
• determining the position the virtual camera on the basis of the display position;
• determining the orientation of the virtual camera on the basis of the display position and the position of the local participant;
• rendering a virtual view of the local participant as if viewed by the virtual camera;
• receiving a further local participant image or a sequence of further local participant images showing the local participant from a different perspective than the view of the local participant;
• and
• analyzing the view of the local participant and the further local participant image or the sequence of further local participant images with respect to 3D information regarding the local participant.

A method for video communication comprising:

• receiving an inbound image or a sequence of inbound images;
• analyzing the inbound image or the sequence of inbound images;
• estimating a location of a sought image feature within the inbound image or the sequence of inbound images;
• determining a display position of the sought image feature when the inbound image or the sequence of inbound images is displayed on a display for a local participant of a video communication;
• receiving a local participant image or a sequence of local participant images depicting at least a part of the local participant;
• determining a position of the local participant on the basis of the local participant image or the sequence of local participant images;
• providing a virtual camera pose on basis of the display position and the position of the local participant;
• rendering a virtual view showing at least the local participant on the basis of the local participant image or the sequence of local participant images and the virtual camera pose, so that the virtual view shows the local participant from a perspective according to the virtual camera pose;
• receiving a further local participant image or a sequence of further local participant images showing the local participant from a different perspective than the local participant image or the sequence of local participant images;
• and
• performing a 3D reconstruction using the local participant image or the sequence of local participant images and the further local participant image or the sequence of further local participant images with respect to 3D information regarding the local participant.

A non-transitory computer readable digital storage medium having stored thereon a computer program comprising a program code for performing, when running on a computer, a method for video communication, the method comprising:

• receiving an inbound image or a sequence of inbound images;
• analyzing the inbound image or the sequence of inbound images;
• estimating a location of a sought image feature within the inbound image or the sequence of inbound images;
• determining a display position of the sought image feature when the inbound image or the sequence of inbound images is displayed on a display for a local participant of a video communication;
• receiving a local participant image or a sequence of local participant images depicting at least a part of the local participant;
• determining a position of the local participant on the basis of the local participant image or the sequence of local participant images;
• providing a virtual camera pose on basis of the display position and the position of the local participant;
• rendering a virtual view showing at least the local participant on the basis of the local participant image or the sequence of local participant images and the virtual camera pose, so that the virtual view shows the local participant from a perspective according to the virtual camera pose;
• receiving a further local participant image or a sequence of further local participant images showing the local participant from a different perspective than the local participant image or the sequence of local participant images;
• and
• performing a 3D reconstruction using the local participant image or the sequence of local participant images and the further local participant image or the sequence of further local participant images with respect to 3D information regarding the local participant.