# US 10,499,992 B2 · Method and system for estimating a deflated lung shape for video assisted thoracic surgery in augmented and mixed reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,499,992 B2
- Document ID: US10499992B2
- Status: Granted
- Kind: B2
- Application number: 15/883,921
- Filed: 2018-01-30
- Published: 2018-06-14
- Granted: 2019-12-10
- Earliest claimed: 2014-10-30
- Assignees: Edda Technology, Inc.
- Applicants: Edda Technology, Inc.
- Inventors: Guo-Qing Wei; Li Fan; Cheng-Chung Liang; Jianzhong Qian; Xiaolan Zeng
- Web page: https://patentgrep.com/patent/US10499992B2

## CPC

- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/62: of area, perimeter, diameter or volume
- A61B 6/032: {Transmission computed tomography [CT]}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30061: Lung
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present teaching relates to surgical procedure assistance. In one example, a first volume of air inside a lung is obtained based on a first image of the lung captured prior to a surgical procedure. The lung has a first shape on the first image. A second volume of air deflated from the lung is determined based on a second image of the lung captured during the surgical procedure. A second shape of the lung is estimated based on the first shape of the lung and the first air volume inside the lung and second volume of air deflated from the lung. A surgical plan is updated based on the estimated second shape of the lung.

## Claims

### Claim 1

A method, implemented on a computing device having at least one processor, storage, and a communication platform capable of connecting to a network for surgical procedure assistance, comprising:
obtaining a first volume of air inside a lung of a patient based on a first image of the lung captured prior to a surgical procedure, wherein the lung has a first shape on the first image;
capturing a video image of the lung during the surgical procedure;
generating a model to estimate a deflation parameter based on a plurality of laparoscopic images;
computing, via the generated model and the video image, a second volume of air deflated from the lung based on the deflation parameter;
estimating a second shape of the lung based on the first shape of the lung, the first air volume inside the lung and second volume of air deflated from the lung;
updating a surgical plan based on the estimated second shape of the lung; and
registering, the estimated second shape of the lung on to the patient in a mixed reality view.

### Claim 2 (depends on 1)

The method of claim 1, wherein the second shape being estimated by condensing a plurality of voxels in the first image along a gravity direction.

### Claim 3 (depends on 1)

The method of claim 1, wherein the surgical procedure includes a video assisted thoracic surgery (VATS), and the first image includes a computed tomography (CT) image.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
obtaining an angle of rotation of the lung from a first position at which the first image was captured to a second position at which the video image was captured; and
adjusting the first shape of the lung on the first image based on the angle of rotation, wherein the second shape of the lung is estimated based on the adjusted first shape of the lung, the first volume, and the second volume.

### Claim 5 (depends on 1)

The method of claim 1, wherein computing the second volume of the air comprises:
generating the plurality of laparoscopic images, wherein each laparoscopic image is generated as a function of a deflation control parameter and a camera position;
training the model to learn a mapping of each laparoscopic image to the deflation control parameter and camera position; and
determining, based on the training, the deflation parameter associated with the video image.

### Claim 6 (depends on 1)

The method of claim 1, wherein estimating the second shape of the lung comprises:
calculating a third volume of air inside the lung during the surgical procedure based on the second volume; and
determining a ratio between the first volume and the third volume.

### Claim 7 (depends on 6)

The method of claim 6, wherein estimating the second shape of the lung further comprises:
estimating, for each voxel of the plurality of voxels included in the first shape of the lung on the first image, the first volume of air inside each voxel based on an intensity of each voxel;
calculating the second volume of air inside each voxel based on the first volume of air inside the voxel and the ratio; and
condensing the plurality of voxels along the gravity direction based on the second volume of air inside each voxel.

### Claim 8

A system for surgical procedure assistance, comprising:
at least one processor configured to:
obtain a first volume of air inside a lung of a patient based on a first image of the lung captured prior to a surgical procedure, wherein the lung has a first shape on the first image;
capture a video image of the lung during the surgical procedure;
generate a model to estimate a deflation parameter based on a plurality of laparoscopic images;
compute, via the generated model and the video image, a second volume of air deflated from the lung based on the deflation parameter;
estimate a second shape of the lung based on the first shape of the lung, the first air volume inside the lung, and second volume of air deflated from the lung;
update a surgical plan based on the estimated second shape of the lung; and
register, the estimated second shape of the lung on to the patient in a mixed reality view.

### Claim 9 (depends on 8)

The system of claim 8, wherein the second shape being estimated by condensing a plurality of voxels in the first image along a gravity direction.

### Claim 10 (depends on 8)

The system of claim 8, wherein the surgical procedure includes a video assisted thoracic surgery (VATS), and the first image includes a computed tomography (CT) image.

### Claim 11 (depends on 8)

The system of claim 8, wherein the at least one processor is further configured to:
obtain an angle of rotation of the lung from a first position at which the first image was captured to a second position at which the video image was captured; and
adjust the first shape of the lung on the first image based on the angle of rotation, wherein the second shape of the lung is estimated based on the adjusted first shape of the lung, the first volume, and the second volume.

### Claim 12 (depends on 8)

The system of claim 8, wherein the at least one processor is further configured to:
generate the plurality of laparoscopic images, wherein each laparoscopic image is generated as a function of a deflation control parameter and a camera position;
train the model to learn a mapping of each laparoscopic image to the deflation control parameter and camera position; and
determine, based on the training, the deflation parameter associated with the video image.

### Claim 13 (depends on 8)

The system of claim 8, wherein the at least one processor is further configured to:
calculate a third volume of air inside the lung during the surgical procedure based on the second volume; and
determine a ratio between the first volume and the third volume.

### Claim 14 (depends on 13)

The system of claim 13, wherein the at least one processor is further configured to:
estimate, for each voxel of the plurality of voxels included in the first shape of the lung on the first image, the first volume of air inside each voxel based on an intensity of each voxel;
calculate the second volume of air inside each voxel based on the first volume of air inside the voxel and the ratio; and
condense the plurality of voxels along the gravity direction based on the second volume of air inside each voxel.