# US 2023/0267675 A1 · 3D Conversations in an Artificial Reality Environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267675 A1
- Document ID: US20230267675A1
- Status: Published application
- Kind: A1
- Application number: 18/311,276
- Filed: 2023-05-03
- Published: 2023-08-24
- Earliest claimed: 2020-07-21
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Brian Keith CABRAL; Albert PARRA POZO
- Web page: https://patentgrep.com/patent/US20230267675A1

## CPC

- G06T 15/20: Perspective computation
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- H04N 2007/145: {Handheld terminals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A 3D conversation system can facilitate 3D conversations in an augmented reality environment, allowing conversation participants to appear as if they are face-to-face. The 3D conversation system can accomplish this with a pipeline of data processing stages, which can include calibrate, capture, tag and filter, compress, decompress, reconstruct, render, and display stages. Generally, the pipeline can capture images of the sending user, create intermediate representations, transform the representations to convert from the orientation the images were taken from to a viewpoint of the receiving user, and output images of the sending user, from the viewpoint of the receiving user, in synchronization with audio captured from the sending user. Such a 3D conversation can take place between two or more sender/receiving systems and, in some implementations can be mediated by one or more server systems. In various configurations, stages of the pipeline can be customized based on a conversation context.

## Claims

### Claim 1

A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for providing a 3D conversation, the process comprising:
obtaining 3D conversation data comprising color information and depth information;
obtaining calibration data comprising one or more of: A) first parameters specifying surrounding conditions and/or context detected for one or more cameras that captured the color information and/or depth information, B) second parameters specifying intrinsic features of the one or more cameras that captured the color information and/or depth information, C) image noise characteristics specifying variations across a sequence of images in the color information and/or depth information, or any combination thereof; and
providing transmission data, based on the 3D conversation data and calibration data, to a reconstruction system that:
transforms, based on the calibration data, a version of the transmission data into a 3D representation by X) combining data, captured by multiple sources, from the version of the transmission data and/or Y) determining position and contour information in 3D space for aspects of the transmission data;
wherein the 3D representation is rendered into one or more 2D images; and
wherein the one or more 2D images are output by a recipient system engaged in the 3D conversation.

### Claim 2 (depends on 1)

The computer-readable storage medium of claim 1, wherein the reconstruction system transforms the version of the transmission data into the 3D representation in part by both X) combining data, captured by multiple sources, from the version of the transmission data and Y) determining position and contour information in 3D space for aspects of the transmission data.

### Claim 3 (depends on 1)

The computer-readable storage medium of claim 1,
wherein the reconstruction system transforms the version of the transmission data into the 3D representation in part by applying one or more transformations to each pixel, specified in the depth information and that corresponds to a depiction of a user, into a 3D representation of at least part of the user based on the calibration data; and
wherein the one or more transformations generate, for each pixel the one or more transformations are applied to, a point in 3D space, wherein that point in 3D space represents a point on the surface of the user and is determined based on a determined camera location specified in the calibration data.

### Claim 4 (depends on 1)

The computer-readable storage medium of claim 1, wherein the reconstruction system applies shading and/or color data onto the 3D representation, using the calibration data to map portions of the color information to the 3D representation.

### Claim 5 (depends on 1)

The computer-readable storage medium of claim 1, wherein the 3D conversation data was acquired from multiple depth sensors and wherein the process further comprises combining the data from the multiple depth sensors into the depth information.

### Claim 6 (depends on 5)

The computer-readable storage medium of claim 5, wherein the combining the data from the multiple depth sensors comprises compensating for distances between the multiple depth sensors by applying, based on the calibration data, a transformation to the data from one or more of the multiple depth sensors.

### Claim 7 (depends on 1)

The computer-readable storage medium of claim 1,
wherein the calibration data comprises the image noise characteristics are in the form of one or more of: photon shot noise, photo response non-uniformity (PRNU), thermal noise, dark current or any combination thereof; and
wherein the image noise characteristics are the result of a statistical analysis of variations across images captured by one of the one or more cameras.

### Claim 8 (depends on 1)

The computer-readable storage medium of claim 1,
wherein the calibration data comprises the image noise characteristics; and
wherein the process further comprises performing noise reduction based on characterizing one or more types of noise in the 3D conversation data by applying shot noise and/or dark current noise to the depth information.

### Claim 9 (depends on 1)

The computer-readable storage medium of claim 1, wherein the 3D representation is one or more of: a point cloud, a signed distance function, populated voxels, a mesh, a light field or any combination thereof.

### Claim 10 (depends on 1)

The computer-readable storage medium of claim 1,
wherein the calibration data comprises the second parameters;
wherein the second parameters include one or more of: focal length, a relationship between a pixel coordinates, lens geometric distortion, or any combination thereof;
wherein the second parameters characterize the optical, geometric, and/or digital characteristics of at least one of the one or more cameras; and
wherein the second parameters are used in the transformation to map between camera coordinates and pixel coordinates of one or more images.

### Claim 11 (depends on 1)

The computer-readable storage medium of claim 1,
wherein the calibration data comprises the first parameters;
wherein the first parameters include one or more of: the location and orientation of at least one of the one or more cameras, one or more heat values, one or more moisture values, lighting characteristics, or any combination thereof;
wherein the first parameters characterize light received at each of multiple camera pixels; and
wherein the first parameters are used in the transformation to interpret light factors in terms of a 3D environment.

### Claim 12 (depends on 1)

The computer-readable storage medium of claim 1, wherein the calibration data comprises one or more of: measurements of camera components, one or more captured images of a pre-defined target, heat data, moisture data, geographic mapping data for the one or more cameras, time-of-flight measurements, lighting conditions, or any combination thereof.

### Claim 13

A method comprising:
obtaining 3D conversation data comprising color information and depth information;
obtaining calibration data comprising A) first parameters specifying surrounding conditions and/or context detected for one or more cameras that captured the color information and/or depth information and/or B) second parameters specifying intrinsic features of the one or more cameras that captured the color information and/or depth information; and
providing transmission data, based on the 3D conversation data and calibration data, to a reconstruction system that:
transforms, based on the calibration data, a version of the transmission data into a 3D representation;
wherein the 3D representation is rendered into one or more 2D images; and
wherein the one or more 2D images are output by a recipient system engaged in the 3D conversation.

### Claim 14 (depends on 13)

The method of claim 13 further comprising:
assigning capture device identifiers to parts of the 3D conversation data according to the device that captured that part of the 3D conversation data, wherein the calibration data is associated with a camera identifier for which the calibration data was generated; and
correlating the calibration data with the at least part of the 3D conversation data based on a match between A) the capture device identifier assigned to the at least part of the 3D conversation data and B) the camera identifier.

### Claim 15 (depends on 13)

The method of claim 13, wherein the output of the one or more 2D images includes a wearable projection system projecting light, based on the one or more 2D images, into at least one eye of a user of the recipient system.

### Claim 16 (depends on 13)

The method of claim 13, wherein the transforming the version of the transmission data into the 3D representation is performed by X) combining data, captured by multiple sources, from the version of the transmission data and/or Y) determining position and contour information in 3D space for aspects of the transmission data.

### Claim 17

A computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
obtaining 3D conversation data comprising color information and depth information;
obtaining calibration data comprising A) first parameters specifying surrounding conditions and/or context detected for one or more cameras that captured the color information and/or depth information and/or B) second parameters specifying intrinsic features of the one or more cameras that captured the color information and/or depth information; and
providing transmission data, based on the 3D conversation data and calibration data, to a reconstruction system that:
transforms, based on the calibration data, a version of the transmission data into a 3D representation;
wherein the 3D representation is rendered into one or more 2D images; and
wherein the one or more 2D images are output by a recipient system engaged in the 3D conversation.

### Claim 18 (depends on 17)

The computing system of claim 17, wherein the transforming the version of the transmission data into the 3D representation is performed by X) combining data, captured by multiple sources, from the version of the transmission data and/or Y) determining position and contour information in 3D space for aspects of the transmission data.

### Claim 19 (depends on 17)

The computing system of claim 17,
wherein the reconstruction system transforms the version of the transmission data into the 3D representation in part by applying one or more transformations to each pixel, specified in the depth information and that corresponds to a depiction of a user, into a 3D representation of at least part of the user based on the calibration data; and
wherein the one or more transformations generate, for each pixel the one or more transformations are applied to, a point in 3D space, wherein that point in 3D space represents a point on the surface of the user and is determined based on a determined camera location specified in the calibration data.

### Claim 20 (depends on 17)

The computing system of claim 17, wherein the reconstruction system applies shading and/or color data onto the 3D representation, using the calibration data to map portions of the color information to the 3D representation.
