# US 2019/0320135 A1 · Telepresence device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0320135 A1
- Document ID: US20190320135A1
- Status: Published application
- Kind: A1
- Application number: 15/955,669
- Filed: 2018-04-17
- Published: 2019-10-17
- Earliest claimed: 2018-04-17
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Ross Garrett CUTLER
- Web page: https://patentgrep.com/patent/US20190320135A1

## CPC

- H04N 7/142: {Constructional details of the terminal equipment, e.g. arrangements of the camera and the display}
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A telepresence device is disclosed, which includes a display and a camera array including cameras configured to capture images of a local subject through the display and output a plurality of local captured images. The telepresence device also includes a local tile selector configured to select a plurality of tiles from the plurality of local captured images based on remote subject point of view (POV) data received from a remote device. The local tile selector outputs local image data that comprises the plurality of selected tiles. The local image data is sent to the remote device to composite an image of the local subject.

## Claims

### Claim 1

A telepresence device comprising:
a display;
a camera array comprising a plurality of cameras;
a processor; and
a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to control the telepresence device to perform functions of:
capturing, using the camera array, a local subject to generate a plurality of local subject capture images;
receiving, via a communication network from a remote device, remote subject point of view (POV) data comprising an estimated future POV of a remote subject;
based on the received remote subject POV data, selecting, from the plurality of local subject capture images, a plurality of candidate local tiles for further selection by the remote device based on the remote subject POV data; and
sending, via the communication network to the remote device, local image data comprising the plurality of candidate local tiles.

### Claim 2 (depends on 1)

The telepresence device of claim 1, wherein the plurality of cameras comprises a light field camera.

### Claim 3 (depends on 1)

The telepresence device of claim 1, wherein the plurality of cameras comprises a depth camera configured to determine a depth estimate between the camera array and the local subject.

### Claim 4 (depends on 1)

The telepresence device of claim 1, wherein instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of estimating a depth between the camera array and the local subject.

### Claim 5 (depends on 1)

The telepresence device of claim 1, wherein the remote subject POV data further comprises a previous or current POV of the remote user.

### Claim 6 (depends on 1)

The telepresence device of claim 1, wherein the camera array comprises a camera module including the camera and an image processor.

### Claim 7 (depends on 1)

The telepresence device of claim 1, wherein instructions, when executed by the processor, further cause the processor to control the telepresence device to perform functions of:
tracking a POV of the local subject; and
sending local subject POV data to the remote device via the communication network, the local subject POV data comprising at least one of previous, current and estimated future POVs of the local subject.

### Claim 8

A telepresence device comprising:
a display;
a camera array comprising a plurality of cameras;
a processor; and
a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to control the telepresence device to perform functions of:
tracking, using the camera array, a point of view (POV) of a local subject;
generating, based on the POV of the local subject, local subject POV data comprising an estimated future POV of the local subject;
sending, via a communication network to a remote device, the local subject POV data;
receiving, via the communication network from the remote device, remote image data comprising a plurality of candidate remote tiles selected by the remote device based the local subject POV data;
based on the local subject POV data, selecting, from the plurality of candidate remote tiles, a plurality of remote tiles;
compositing an image of a remote subject based on the plurality of selected remote tiles; and
displaying, on the display, the composited image of the remote subject.

### Claim 9 (depends on 8)

The telepresence device of claim 8, wherein the plurality of cameras comprises a light field camera.

### Claim 10 (depends on 8)

The telepresence device of claim 8, wherein the camera array comprises a camera module including the camera and an image sensor.

### Claim 11 (depends on 8)

The telepresence device of claim 8, wherein the local subject POV data further comprises a previous or current POV of the local subject.

### Claim 12 (depends on 8)

The telepresence device of claim 8, wherein the remote image data includes a depth estimate between the remote device and the remote subject.

### Claim 13

(canceled)

### Claim 14

A telepresence device comprising:
a display;
a camera array comprising a plurality of cameras;
a processor; and
a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to control the telepresence device to perform functions of:
capturing, using the camera array, a local subject to generate a plurality of local subject capture images;
receiving, via a communication network form the remote device, remote subject point of view (POV) data comprising an estimated future POV of a remote subject;
based on the remote subject POV data, selecting, from the plurality of local subject capture images, a plurality of candidate local tiles for further selection by the remote device based on the remote subject POV data;
sending, via a communication network to the remote device, local image data comprising the plurality of candidate local tiles;
tracking a POV of the local subject and generating local subject POV data;
sending, via the communication network to the remote device, the local POV data comprising an estimate future POV of the local subject;
receiving, via the communication network from the remote device, remote image data comprising a plurality of candidate remote tiles selected by the remote device based on the local POV data;
based on the local subject POV data, selecting, from the plurality of candidate remote tiles, a plurality of remote tiles;
compositing an image of the remote subject based on the plurality of selected remote tiles; and
displaying the composited image of the remote subject on the display.

### Claim 15 (depends on 14)

The telepresence device of claim 14, wherein the plurality of cameras comprises a light field camera.

### Claim 16 (depends on 14)

The telepresence device of claim 14, wherein the plurality of cameras includes a depth camera configured to determine a depth estimate between the camera array and the local subject.

### Claim 17 (depends on 14)

The telepresence device of claim 14, wherein the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of estimating a depth between the camera array and the local subject, wherein the local image data comprises the estimated depth.

### Claim 18 (depends on 14)

The telepresence device of claim 14, wherein the camera array comprises a camera module including the camera and an image processor.

### Claim 19

(canceled)

### Claim 20 (depends on 14)

The telepresence device of claim 14, wherein:
the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of sending, via the communication network to the remote device, the local POV data, and
the plurality of candidate remote tiles is selected, based on the local POV data, by the remote device based on the local POV data for further selection by the telepresence device.

### Claim at position 21

21-30. (canceled)

### Claim 31

A method of operating a telepresence device comprising a camera array, the method comprising:
capturing, using the camera array, a local subject to generate a plurality of local subject capture images;
receiving, via a communication network from a remote device, remote subject point of view (POV) data comprising an estimated future POV of a remote subject;
based on the received remote subject POV data, selecting, from the plurality of local subject capture images, a plurality of candidate local tiles for further selection by the remote device based on the remote subject POV data; and
sending, via the communication network to the remote device, local image data comprising the plurality of candidate local tiles.

### Claim 32 (depends on 31)

The method of claim 31, further comprising estimating a depth between the camera array and the local subject.

### Claim 33 (depends on 32)

The method of claim 32, wherein the local image data further comprises the estimated depth between the camera array and the local subject.

### Claim 34 (depends on 31)

The method of claim 31, further comprising:
tracking a POV of the local subject; and
sending local subject POV data to the remote device, the local subject POV data comprising an estimated future POV of the local subject.

### Claim 35

A method of operating a telepresence device comprising a display and camera array, the method comprising:
tracking, using the camera array, a point of view (POV) of a local subject;
generating, based on the POV of the local subject, local subject POV data comprising an estimated future POV of the local subject;
sending, via a communication network to a remote device, the local subject POV data;
receiving, via the communication network from the remote device, remote image data comprising a plurality of candidate remote tiles selected by the remote device based the local subject POV data;
based on the local subject POV data, selecting, from the plurality of candidate remote tiles, a plurality of remote tiles;
compositing an image of a remote subject based on the plurality of remote tiles; and
displaying, on the display, the composited image of the remote subject.

### Claim 36 (depends on 35)

The method of claim 35, wherein the remote image data includes a depth estimate between the remote device and the remote subject.

### Claim 37

(canceled)

### Claim 38 (depends on 35)

The method of claim 35, further comprising:
capturing, using the camera array, the local subject to generate a plurality of local subject capture images;
receiving, via the communication network from the remote device, remote subject POV data comprising an estimated future POV of a remote subject;
based on the received remote subject POV data, selecting, from the plurality of local subject capture images, a plurality of candidate local tiles; and
sending, via the communication network to the remove device, local image data comprising the plurality of candidate local tiles for further selection by the remote device based on the remote subject POV data.

### Claim 39 (depends on 1)

The telepresence device of claim 1, wherein, for selecting the plurality of candidate local tiles, the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of selecting, based a first estimate future POV of the remote subject, the plurality of candidate local tiles for further selection by the remote device based on a second estimate future POV of the remote subject.

### Claim 40 (depends on 8)

The telepresence device of claim 8, wherein:
the received remote image data comprises the plurality of candidate remote tiles selected by the remote device based on a first estimated future POV of the local subject, and
for selecting the plurality of remote tiles, the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of selecting, from the plurality of candidate remote tiles, the plurality of remote tiled based on a second estimated future POV of the local subject.

### Claim 41 (depends on 14)

The telepresence device of claim 14, wherein:
for selecting the plurality of candidate local tiles, the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of selecting, based a first estimate future POV of the remote subject, the plurality of candidate local tiles for further selection by the remote device based on a second estimate future POV of the remote subject,
the received remote image data comprises the plurality of candidate remote tiles selected by the remote device based on a first estimated future POV of the local subject, and
for selecting the plurality of remote tiles, the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of selecting, from the plurality of candidate remote tiles, the plurality of remote tiled based on a second estimated future POV of the local subject.

### Claim 42 (depends on 31)

The method of claim 31, wherein selecting the plurality of candidate local tiles comprises selecting, based a first estimate future POV of the remote subject, the plurality of candidate local tiles for further selection by the remote device based on a second estimate future POV of the remote subject.

### Claim 43 (depends on 35)

The method of claim 35, wherein the received remote image data comprises the plurality of candidate remote tiles selected by the remote device based on a first estimated future POV of the local subject, and
selecting the plurality of remote tiles comprises selecting, from the plurality of candidate remote tiles, the plurality of remote tiled based on a second estimated future POV of the local subject.
