US 11,451,746 B1Granted

Image and audio data processing to create mutual presence in a video conference

AssigneeAmazon Technologies, Inc.

InventorsJules C. Graybill, Katherine M. Ryan, Maren Marie Costa, Marshall Friend Tappen, Kristina Lisa Klinkner, Gregory J. Bartlett, Robert Wadia Moses

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Systems and methods to improve video conferencing may include various improvements and modifications to image and audio data to simulate mutual presence among users. For example, automated positioning and presentation of image data of users via presentation devices may present users at a position, size, and scale to simulate mutual presence of users within a same physical space. In addition, rendering image data of users from virtual camera positions via presentation devices may simulate mutual eye gaze and mutual presence of users. Further, blending a background, or accentuating a user, within image data that is presented via presentation devices may match visual characteristics of the environment or the user to simulate mutual presence of users. Moreover, dynamic steering of individual audio transducers of an audio transducer array, and simulating audio point sources using a combination of speakers, may simulate direct communication and mutual presence of users.

Claims20

Dependent claims

A video conference system to facilitate communications between a local environment and a remote environment, comprising:

a first presentation device and a first, wide angle, high resolution imaging device associated with the local environment;
a second presentation device and a second, wide angle, high resolution imaging device associated with the remote environment;
and
a controller configured to at least:
receive, from the first imaging device associated with the local environment, image data of the local environment of the video conference system;
process the image data of the local environment to identify a first user within the local environment;
perform segmentation of the image data associated with the first user to identify body parts associated with the first user;
receive, from the second imaging device associated with the remote environment, image data of the remote environment of the video conference system;
process the image data of the remote environment to identify a second user within the remote environment;
perform segmentation of the image data associated with the second user to identify body parts associated with the second user;
determine associations between respective regions of the second presentation device and respective body parts associated with the first user based at least in part on positions of respective body parts associated with the second user relative to the second presentation device;
modify the image data associated with the first user to align the respective body parts associated with the first user with the respective regions of the second presentation device;
present, via the second presentation device to the second user within the remote environment, the modified image data associated with the first user;
determine associations between respective regions of the first presentation device and respective body parts associated with the second user based at least in part on positions of respective body parts associated with the first user relative to the first presentation device;
modify the image data associated with the second user to align the respective body parts associated with the second user with the respective regions of the first presentation device;
and
present, via the first presentation device to the first user within the local environment, the modified image data associated with the second user.

14 more elements

A video conference system to facilitate communications between a local environment and a remote environment, comprising:

a first presentation device and a first, wide angle, high resolution imaging device associated with the local environment;
a first left side imaging device associated with a left side of the first presentation device within the local environment, and a first right side imaging device associated with a right side of the first presentation device within the local environment, the first left side imaging device and the first right side imaging device being aligned with each other along a substantially horizontal line;
a second presentation device and a second, wide angle, high resolution imaging device associated with the remote environment;
a second left side imaging device associated with a left side of the second presentation device within the remote environment, and a second right side imaging device associated with a right side of the second presentation device within the remote environment, the second left side imaging device and the second right side imaging device being aligned with each other along a substantially horizontal line;
and
a controller configured to at least:
receive, from the first left side imaging device associated with the local environment, left side image data of the local environment of the video conference system;
receive, from the first right side imaging device associated with the local environment, right side image data of the local environment of the video conference system;
process the left side image data of the local environment to identify first left side pixels associated with a first user within the local environment;
process the right side image data of the local environment to identify first right side pixels associated with the first user within the local environment;
determine first matching pixels between the first left side pixels and the first right side pixels associated with the first user within the local environment;
determine disparities between the first matching pixels from the first left side pixels and the first right side pixels associated with the first user;
determine distances of the first matching pixels from each of the first left side imaging device and the first right side imaging device based at least in part on the determined disparities;
determine three-dimensional locations of the first matching pixels relative to the first left side imaging device and the first right side imaging device based at least in part on the determined distances;
shift the first matching pixels to respective locations associated with a first virtual imaging device positioned between the first left side imaging device and the first right side imaging device based at least in part on the determined three-dimensional locations of the first matching pixels;
present, via the second presentation device to a second user within the remote environment, the first matching pixels shifted to respective locations associated with the first virtual imaging device in combination with at least a portion of image data from the first high resolution imaging device;
receive, from the second left side imaging device associated with the remote environment, left side image data of the remote environment of the video conference system;
receive, from the second right side imaging device associated with the remote environment, right side image data of the remote environment of the video conference system;
process the left side image data of the remote environment to identify second left side pixels associated with the second user within the remote environment;
process the right side image data of the remote environment to identify second right side pixels associated with the second user within the remote environment;
determine second matching pixels between the second left side pixels and the second right side pixels associated with the second user within the remote environment;
determine disparities between the second matching pixels from the second left side pixels and the second right side pixels associated with the second user;
determine distances of the second matching pixels from each of the second left side imaging device and the second right side imaging device based at least in part on the determined disparities;
determine three-dimensional locations of the second matching pixels relative to the second left side imaging device and the second right side imaging device based at least in part on the determined distances;
shift the second matching pixels to respective locations associated with a second virtual imaging device positioned between the second left side imaging device and the second right side imaging device based at least in part on the determined three-dimensional locations of the second matching pixels;
and
present, via the first presentation device to the first user within the local environment, the second matching pixels shifted to respective locations associated with the second virtual imaging device in combination with at least a portion of image data from the second high resolution imaging device.

24 more elements

A computer-implemented method to operate a video conference system, comprising:

receiving, by a controller from an imaging device within a remote environment, image data of a second user within the remote environment of the video conference system;
performing, by the controller, segmentation of the image data of the second user to identify body parts associated with the second user;
receiving, by the controller from an imaging device within a local environment, image data of a first user within the local environment of the video conference system;
performing, by the controller, segmentation of the image data of the first user to identify positions of body parts associated with the first user relative to a first presentation device;
determining, by the controller, associations between respective regions of the first presentation device within the local environment and respective body parts associated with the second user based at least in part on respective positions of body parts associated with the first user relative to the first presentation device;
modifying, by the controller, the image data of the second user to align the respective body parts associated with the second user with the respective regions of the first presentation device;
and
presenting, by the controller, via the first presentation device to the first user within the local environment, the modified image data of the second user.

5 more elements

A computer-implemented method to operate a video conference system, comprising:

receiving, by a controller from an imaging device within a remote environment, image data of a second user within the remote environment of the video conference system, wherein the imaging device within the remote environment comprises a high resolution imaging device;
receiving, by the controller from a left side imaging device within the remote environment, left side image data of the second user within the remote environment;
receiving, by the controller from a right side imaging device within the remote environment, right side image data of the second user within the remote environment;
processing, by the controller, the left side image data to identify left side pixels associated with the second user;
processing, by the controller, the right side image data to identify right side pixels associated with the second user;
determining, by the controller, disparities between matching pixels from the left side pixels and the right side pixels associated with the second user;
determining, by the controller, three-dimensional locations of the matching pixels relative to the left side imaging device and the right side imaging device based at least in part on the determined disparities;
shifting, by the controller, the matching pixels to respective locations associated with a virtual imaging device positioned between the left side imaging device and the right side imaging device based at least in part on the determined three-dimensional locations of the matching pixels;
and
presenting, by the controller via a first presentation device to a first user within a local environment, the matching pixels shifted to respective locations associated with the virtual imaging device in combination with at least a portion of the image data from the high resolution imaging device.

7 more elements

A video conference system to facilitate communications between a local environment and a remote environment, comprising:

a first presentation device and a first, wide angle, high resolution imaging device associated with the local environment;
a second, wide angle, high resolution imaging device associated with the remote environment;
and
a controller configured to at least:
receive, from the second imaging device associated with the remote environment, image data of a second user within the remote environment of the video conference system;
perform segmentation of the image data associated with the second user to identify body parts associated with the second user;
receive, from the first imaging device within the local environment, image data of a first user within the local environment of the video conference system;
perform segmentation of the image data of the first user to identify positions of body parts associated with the first user relative to the first presentation device;
determine associations between respective regions of the first presentation device and respective body parts associated with the second user based at least in part on respective positions of body parts associated with the first user relative to the first presentation device;
modify the image data associated with the second user to align the respective body parts associated with the second user with the respective regions of the first presentation device;
and
present, via the first presentation device to the first user within the local environment, the modified image data associated with the second user.

9 more elements

A video conference system to facilitate communications between a local environment and a remote environment, comprising:

a first presentation device and a first, wide angle, high resolution imaging device associated with the local environment;
a second, wide angle, high resolution imaging device associated with the remote environment;
and
a controller configured to at least:
receive, from the second high resolution imaging device associated with the remote environment, image data of a second user within the remote environment of the video conference system;
receive, from a second left side imaging device associated with the remote environment, left side image data of the second user within the remote environment;
receive, from a second right side imaging device associated with the remote environment, right side image data of the second user within the remote environment;
process the left side image data to identify left side pixels associated with the second user within the remote environment;
process the right side image data to identify right side pixels associated with the second user within the remote environment;
determine disparities between matching pixels from the left side pixels and the right side pixels associated with the second user;
determine three-dimensional locations of the matching pixels relative to the second left side imaging device and the second right side imaging device based at least in part on the determined disparities;
shift the matching pixels to respective locations associated with a virtual imaging device positioned between the second left side imaging device and the second right side imaging device based at least in part on the determined three-dimensional locations of the matching pixels;
and
present, via the first presentation device to a first user within the local environment, the matching pixels shifted to respective locations associated with the virtual imaging device in combination with at least a portion of the image data from the second high resolution imaging device.

11 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/204,145Filed 17 Mar 2021 · us provisional application
Priority: 26 Mar 2020
Granted: 20 Sep 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification19 codes

G06T 3/00
Geometric image transformations in the plane of the image
H04N 7/15
Systems for two-way working: conference systems
G06T 3/60
Geometric image transformations in the plane of the image: rotation of whole images or parts thereof
G06T 7/11
Segmentation; Edge detection: region-based segmentation

Family

Application63/000,120Us provisional application · filed 26 Mar 2020
US 11,451,746 B1This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key