# US 2022/0012954 A1 · Generation of synthetic three-dimensional imaging from partial depth maps

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0012954 A1
- Document ID: US20220012954A1
- Status: Published application
- Kind: A1
- Application number: 17/349,713
- Filed: 2021-06-16
- Published: 2022-01-13
- Earliest claimed: 2018-12-28
- Assignees: Activ Surgical, Inc.
- Applicants: Activ Surgical, Inc.
- Inventors: Vasiliy E. Buharin
- Web page: https://patentgrep.com/patent/US20220012954A1

## CPC

- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 7/75: {involving models}
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30084: Kidney; Renal
- G06T 2210/41: Medical
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T 2219/2016: Rotation, translation, scaling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Generation of synthetic three-dimensional imaging from partial depth maps is provided. In various embodiments, an image of an anatomical structure is received from a camera. A depth map corresponding to the image is received from a depth sensor that may be a part of the camera or separate from the camera. A preliminary point cloud corresponding to the anatomical structure is generated based on the depth map and the image. The preliminary point cloud is registered with a model of the anatomical structure. An augmented point cloud is generated from the preliminary point cloud and the model. The augmented point cloud is rotated in space. The augmented point cloud is rendered. The rendered augmented point cloud is displayed to a user.

## Claims

### Claim at position 1

-78. (canceled)

### Claim 79

A method comprising:
(a) obtaining a plurality of images of an anatomical structure of a patient and a plurality of depth maps corresponding to the plurality of images;
(b) generating a point cloud corresponding to the anatomical structure using the plurality of images and the plurality of depth maps, wherein each point in the point cloud comprises a depth value derived from the one or more depth maps and a color value derived from the one or more images; and
(c) rendering and displaying the point cloud such that the point cloud is capable of being (i) manipulated by a user in three-dimensional space and (ii) viewed from a plurality of preprogrammed viewpoints.

### Claim 80 (depends on 79)

The method of claim 79, wherein the point cloud is capable of being rotated in three-dimensional space in order to provide a visualization of the anatomical structure from one or more viewpoints of interest.

### Claim 81 (depends on 79)

The method of claim 79, wherein the plurality of preprogrammed viewpoints are tailored for one or more procedures to assist in re-orientating a physician during the one or more procedures.

### Claim 82 (depends on 79)

The method of claim 79, further comprising, subsequent to (a), post-processing the plurality of depth maps to reduce noise in one or more depth frames, wherein post-processing comprises at least one of spatial filtering and temporal filtering.

### Claim 83 (depends on 82)

The method of claim 82, wherein spatial filtering comprises updating one or more depth values using depth data associated with one or more pixels in a single frame.

### Claim 84 (depends on 82)

The method of claim 82, wherein temporal filtering comprises updating one or more depth values using depth data associated with one or more pixels in a plurality of frames.

### Claim 85 (depends on 79)

The method of claim 79, further comprising, subsequent to (b), rotating the point cloud.

### Claim 86 (depends on 79)

The method of claim 79, wherein (b) further comprises generating a plurality of point clouds comprising the point cloud and aligning the plurality of point clouds relative to each other.

### Claim 87 (depends on 86)

The method of claim 86, wherein (c) further comprises rendering and displaying the plurality of aligned point clouds to the user.

### Claim 88 (depends on 86)

The method of claim 86, wherein each of the plurality of points clouds corresponds to different image frames associated with different perspective views of the anatomical structure.

### Claim 89 (depends on 86)

The method of claim 86, wherein aligning the plurality of points clouds comprises registering the points clouds using a rigid body registration or a deformable registration.

### Claim 90 (depends on 79)

The method of claim 79, wherein in (b), the point cloud is generated using data from two or more image frames.

### Claim 91 (depends on 79)

The method of claim 79, wherein the point cloud is a preliminary point cloud, wherein the method further comprises registering the preliminary point cloud with a model of the anatomical structure and generating an augmented point cloud from the preliminary point cloud and the model.

### Claim 92 (depends on 91)

The method of claim 91, further comprising:
receiving from the user an indication to rotate the augmented point cloud;
rotating the augmented point cloud in space according to the indication; and
rendering and displaying the augmented point cloud to the user.

### Claim 93 (depends on 91)

The method of claim 91, further comprising generating a surface mesh from the preliminary point cloud.

### Claim 94 (depends on 93)

The method of claim 93, wherein generating the surface mesh comprises interpolating the preliminary point cloud.

### Claim 95 (depends on 93)

The method of claim 93, further comprising, prior to generating the surface mesh, segmenting the preliminary point cloud into two or more semantic regions.

### Claim 96 (depends on 95)

The method of claim 95, wherein generating the surface mesh comprises generating separate surface meshes for each of the two or more semantic regions.

### Claim 97 (depends on 96)

The method of claim 96, further comprising combining the separate surface meshes into a combined surface mesh and displaying the combined surface mesh to the user.

### Claim 98 (depends on 91)

The method of claim 91, wherein the model of the anatomical structure comprises a virtual three-dimensional (3D) model or an anatomical atlas, wherein the model of the anatomical structure corresponds to at least one of (i) pre-operative imaging of the patient or (ii) a three-dimensional (3D) reconstruction based at least in part on the pre-operative imaging.
