# US 2019/0320142 A1 · Telepresence devices operation methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0320142 A1
- Document ID: US20190320142A1
- Status: Published application
- Kind: A1
- Application number: 15/955,672
- Filed: 2018-04-17
- Published: 2019-10-17
- Earliest claimed: 2018-04-17
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Ross Garrett CUTLER
- Web page: https://patentgrep.com/patent/US20190320142A1

## CPC

- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- H04L 12/1827: {Network arrangements for conference optimisation or adaptation}
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method of operating a telepresence device is disclosed, which includes capturing images of a local subject using a plurality of cameras to generate a plurality of local captured images. Remote subject point of view (POV) data is received from a remote device. A plurality of tiles is selected from the plurality of local captured images based on the remote subject POV data. Local image data is sent to the remote device. The local image data includes the plurality of selected tiles.

## Claims

### Claim 1

A method of operating a telepresence device, comprising:
capturing an image of a local subject using a plurality of cameras to generate a plurality of local subject captured images;
receiving remote subject point of view (POV) data from a remote device via a communication network, the remote subject POV data comprising an estimated future POV of a remote subject;
selecting a plurality of tiles from the plurality of local subject captured images based on the received remote subject POV data;
generating local image data comprising the plurality of selected tiles; and
sending the local image data to the remote device via the communication channel.

### Claim 2 (depends on 1)

The method of claim 1, wherein capturing the local subject comprises capturing, using the plurality of cameras, the local subject through a display.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of cameras comprises a light field camera.

### Claim 4 (depends on 1)

The method of claim 1, further comprising estimating a depth between the plurality of cameras and the local subject.

### Claim 5 (depends on 4)

The method of claim 4, further comprising adding depth information to the local image data, the depth information comprising the estimated depth.

### Claim 6 (depends on 1)

The method of claim 1, wherein the remote subject POV data further comprises a previous or current POV of the remote subject.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
tracking a POV of the local subject to generate local subject POV data; and
sending the local subject POV data to the remote device via the communication network.

### Claim 8 (depends on 7)

The method of claim 7, wherein the local subject POV data comprises at least one of previous, current and estimated future POVs of the local subject.

### Claim 9

A method of operating a telepresence device, comprising:
tracking a point of view (POV) of a local subject;
generating local subject POV data based on the POV of the local subject, the local subject POV data comprising an estimated future POV of the local subject;
receiving remote image data from a remote device via a communication network, the remote image data comprising a plurality of remote tiles;
selecting, based on the local subject POV data, a plurality of tiles from the plurality of remote tiles;
compositing an image of a remote subject based on the plurality of selected tiles to produce a composited image; and
displaying, on a display, the image of the composited image of the remote subject.

### Claim 10 (depends on 9)

The method of claim 9, wherein capturing the local subject comprises capturing, using a plurality of cameras, the local subject through the display.

### Claim 11 (depends on 9)

The method of claim 9, wherein the plurality of cameras comprises a light field camera.

### Claim 12 (depends on 9)

The method of claim 9, wherein the local subject POV data further comprises a previous or current POV of the local subject.

### Claim 13 (depends on 9)

The method of claim 9, further comprising sending the local subject POV data to the remote device via the communication network, wherein the remote device selects the plurality of remote tiles the local subject POV data.

### Claim 14 (depends on 13)

The method of claim 13, wherein the plurality of remote tiles are selected based on the estimated future POV of the local subject.

### Claim 15 (depends on 9)

The method of claim 9, wherein the remote image data comprises depth information of the remote subject, the depth information comprising an estimated depth between the remote subject and the remote device.

### Claim 16

A method of operating telepresence devices, comprising:
capturing an image of a first subject located at a first location, using a plurality of first cameras of a first telepresence device, to generate a plurality of first captured images of the first subject;
capturing an image of a second subject located at a second location, using a plurality of second cameras of a second telepresence device, to generate a plurality of second captured images of the second subject;
processing, at the first telepresence device, the plurality of first captured images to generate first subject point of view (POV) data, the first POV data comprising an estimated future POV of the first subject;
processing, at the second telepresence device, the plurality of second captured images to generate second subject POV data, the second POV data comprising an estimated future POV of the second subject;
sending the first subject POV data from the first telepresence device to the second telepresence device;
sending the second subject POV data from the second telepresence device to the first telepresence device;
selecting, at the first telepresence device, a plurality of first tiles from the plurality of first captured images based on the second subject POV data;
sending first image data from the first telepresence device to the second telepresence device, the first image data comprising the plurality of first tiles;
sending second image data from the second telepresence device to the first telepresence device, the second image data comprising the plurality of second tiles;
selecting, at the first telepresence device, a plurality of third tiles from the plurality of second tiles based on the first subject POV data; and
displaying, on a first display of the first telepresence device, an image of the second subject composited based on the plurality of third tiles.

### Claim 17 (depends on 16)

The method of claim 16, wherein the plurality of first and second cameras comprise a light field camera.

### Claim 18 (depends on 16)

The method of claim 16, further comprising:
selecting, at the second telepresence device, a plurality of fourth tiles from the plurality of first tiles based on the first subject POV data; and
displaying, on a second display of the second telepresence device, an image of the first subject composited based on the plurality of fourth tiles.

### Claim 19 (depends on 18)

The method of claim 18, wherein the plurality of first cameras and second cameras capture the images of the first and second subjects through the first and second displays, respectively.

### Claim 20 (depends on 18)

The method of claim 18, wherein the first subject POV data further comprises a previous or current and estimated future POV of the first subject, and the second subject POV data further comprises a previous or current and estimated future POV of the second subject.

### Claim 21 (depends on 16)

The method of claim 16, further comprising estimating a first depth between the first subject and a plurality of first cameras,
wherein the first tiles first image data comprises the estimated first depth.

### Claim 22 (depends on 16)

The method of claim 16, further comprising estimating a second depth between the second subject and a plurality of second cameras,
wherein the second image data comprises the estimated second depth.

### Claim 23

A non-transient computer readable medium comprising instructions for causing a computer to perform functions of:
capturing an image of a local subject using a plurality of cameras to generate a plurality of local subject captured images;
receiving remote subject point of view (POV) data from a remote device via a communication network, the remote subject POV data comprising an estimated future POV of a remote subject;
selecting a plurality of first tiles from the plurality of local subject captured images based on the received remote subject POV data; and
sending local image data to the remote device, the local image data comprising the plurality of selected first tiles.

### Claim 24 (depends on 23)

The non-transient computer readable medium of claim 23, wherein the plurality of cameras are arranged to capture the local subject through a display.

### Claim 25 (depends on 23)

The non-transient computer readable medium of claim 23, wherein the plurality of cameras comprises a light field camera.

### Claim 26 (depends on 23)

The non-transient computer readable medium of claim 23, further comprising instructions for causing the computer to perform a function of estimating a depth between the plurality of cameras and the local subject.

### Claim 27 (depends on 26)

The non-transient computer readable medium of claim 26, further comprising instructions for causing the computer to perform functions of:
estimating a depth between the local subject and the plurality of cameras; and
adding depth information to the local image data, the depth information comprising the estimated depth.

### Claim 28 (depends on 23)

The non-transient computer readable medium of claim 23, wherein the remote subject POV data further comprises a previous or current of the remote subject.

### Claim 29 (depends on 23)

The non-transient computer readable medium of claim 23, further comprising instructions for causing the computer to perform functions of:
generating local subject POV data;
sending the local subject POV data to the remote device;
receiving remote image data from the remote device, the remote image data comprising a plurality of second tiles selected at the remote device based on the local subject POV data;
selecting a plurality of third tiles from the plurality of second tiles based on the local subject POV data; and
displaying, on a display, the image of the remote subject composited based on the plurality of third tiles.

### Claim 30 (depends on 29)

The non-transient computer readable medium of claim 29, wherein the local subject POV data comprises at least one of previous, current and estimated future POVs of the local subject.

### Claim 31

A method comprising:
capturing an image of a local subject using a plurality of cameras to generate a plurality of local subject captured images;
tracking, using the plurality of cameras, a point of view (POV) of the local subject:
generating local subject POV data based on the POV of the local subject, the local subject POV data comprising an estimated future POV of the local subject;
selecting a plurality of tiles from the local subject captured image; and
converting the plurality of selected tiles to local subject image data that is usable by a remote device to render and display a 3D visual representation of the local subject,
wherein the local subject image data comprises the local subject POV data that is usable by the remote device to maintain eye gazing between a remote device user and the 3D visual representation of the local subject.

### Claim 32 (depends on 31)

The method of claim 31, wherein the 3D visual representation is a 3D avatar of the local subject.
