# US 2024/0203055 A1 · Representing flat surfaces in point-based representations of physical environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203055 A1
- Document ID: US20240203055A1
- Status: Published application
- Kind: A1
- Application number: 18/582,757
- Filed: 2024-02-21
- Published: 2024-06-20
- Earliest claimed: 2021-09-10
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Long H. NGO; Alexandre DA VEIGA
- Web page: https://patentgrep.com/patent/US20240203055A1

## CPC

- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06V 20/64: Three-dimensional [3D] objects
- G06T 2210/04: Architectural design, interior design
- G06T 2210/56: Particle system, point based geometry or rendering
- G06V 2201/12: Acquisition of 3D measurements of objects
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Various implementations disclosed herein include devices, systems, and methods that generate a 3D representation of a physical environment by generating a point cloud and selectively replacing some of the points. For example, points representing flat surfaces (e.g., walls and ceilings) may be replaced with planar elements that are “painted” using image data. In contrast, points representing non-flat portions (e.g. furniture, curtains, wall hangings, etc.) are left as points in the model or altered in a different way. Selectively altering the 3D representation may provide (a) a cleaner feeling and/or (b) a more compact 3D representation for more efficient and faster communication and/or rendering.

## Claims

### Claim 1

A method comprising:
at a processor of a first device:
generating a three-dimensional (3D) representation of a physical environment, the 3D representation comprising points each having a 3D location and representing a portion of the physical environment;
identifying a first set of the points of the 3D representation satisfying a replacement criterion; and
replacing the first set of the points in the 3D representation with a planar element having an appearance provided by one or more images depicting a surface of the physical environment.

### Claim 2 (depends on 1)

The method of claim 1 further comprising providing the 3D representation to a second device different than the first device, wherein the second device provides a view based on the points and the planar element, wherein an appearance of the planar element in the view is based on the one or more images of the surface.

### Claim 3 (depends on 2)

The method of claim 2, wherein the first device and the second device concurrently provide views of the physical environment during a communication session.

### Claim 4 (depends on 1)

The method of claim 1, where the surface is a flat surface, wherein the planar element comprises a mesh defining the flat surface, wherein an appearance of the mesh is defined based on the one or more images.

### Claim 5 (depends on 1)

The method of claim 1, wherein the planar element is defined using less data than the first set of the points.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first set of the points is identified by:
determining a confidence value that each point of the first set of the points represents a portion of a flat surface; and
determining that the confidence value of each point of the first set of the points exceeds a threshold.

### Claim 7 (depends on 6)

The method of claim 6, wherein the first set of the points is further identified based on determining that the flat surface exceeds a size threshold.

### Claim 8 (depends on 1)

The method of claim 1, wherein the replacing occurs in real time during a scanning process in which:
sensor data is obtained over a time period;
active depth sensing and imaging are used to generate the points for inclusion in the 3D representation as sensor data is received during the time period; and
points are selectively replaced with one or more planar elements based on the replacement criterion during the time period.

### Claim 9 (depends on 1)

The method of claim 1 further comprising updating an appearance of the planar element as new sensor data is obtained.

### Claim 10 (depends on 9)

The method of claim 9, wherein updating the appearance of the planar element is based on determining that the new sensor data satisfies an image criterion.

### Claim 11

A first device comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
generating a three-dimensional (3D) representation of a physical environment, the 3D representation comprising points each having a 3D location and representing a portion of the physical environment;
identifying a first set of the points of the 3D representation satisfying a replacement criterion; and
replacing the first set of the points in the 3D representation with a planar element having an appearance provided by one or more images depicting a surface of the physical environment.

### Claim 12 (depends on 11)

The first device of claim 11, wherein the operations further comprise providing the 3D representation to a second device different than the first device, wherein the second device provides a view based on the points and the planar element, wherein an appearance of the planar element in the view is based on the one or more images of the surface.

### Claim 13 (depends on 12)

The first device of claim 12, wherein the first device and the second device concurrently provide views of the physical environment during a communication session.

### Claim 14 (depends on 11)

The first device of claim 11, where the surface is a flat surface, wherein the planar element comprises a mesh defining the flat surface, wherein an appearance of the mesh is defined based on the one or more images.

### Claim 15 (depends on 11)

The first device of claim 11, wherein the planar element is defined using less data than the first set of the points.

### Claim 16 (depends on 11)

The first device of claim 11, wherein the first set of the points is identified by:
determining a confidence value that each point of the first set of the points represents a portion of a flat surface; and
determining that the confidence value of each point of the first set of the points exceeds a threshold.

### Claim 17 (depends on 16)

The first device of claim 16, wherein the first set of the points is further identified based on determining that the flat surface exceeds a size threshold.

### Claim 18 (depends on 1)

The first device of claim 1, wherein the replacing occurs in real time during a scanning process in which:
sensor data is obtained over a time period;
active depth sensing and imaging are used to generate the points for inclusion in the 3D representation as sensor data is received during the time period; and
points are selectively replaced with one or more planar elements based on the replacement criterion during the time period.

### Claim 19 (depends on 1)

The first device of claim 1, wherein the operations further comprise updating an appearance of the planar element as new sensor data is obtained, wherein updating the appearance of the planar element is based on determining that the new sensor data satisfies an image criterion.

### Claim 20

A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
generating a three-dimensional (3D) representation of a physical environment, the 3D representation comprising points each having a 3D location and representing a portion of the physical environment;
identifying a first set of the points of the 3D representation satisfying a replacement criterion; and
replacing the first set of the points in the 3D representation with a planar element having an appearance provided by one or more images depicting a surface of the physical environment.

### Claim at position 21

21-28. (canceled)
