# US 2021/0280312 A1 · Detecting deficient coverage in gastroenterological procedures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0280312 A1
- Document ID: US20210280312A1
- Status: Published application
- Kind: A1
- Application number: 17/187,382
- Filed: 2021-02-26
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: Verily Life Sciences LLC
- Applicants: Verily Life Sciences LLC
- Inventors: Daniel Freedman; Yacob Yochai Blau; Liran Katzir; Amit Aides; Ilan Moshe Shimshoni; Ehud Benyamin Rivlin; Yossi Matias
- Web page: https://patentgrep.com/patent/US20210280312A1

## CPC

- A61B 1/000096: {using artificial intelligence}
- A61B 1/31: for the rectum, e.g. proctoscopes or sigmoidoscopes
- G06T 7/0016: {involving temporal comparison}
- G06T 7/50: Depth or shape recovery
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10068: Endoscopic image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30028: Colon; Small intestine
- G06T 2207/30032: Colon polyp
- G06V 2201/031: of internal organs
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

The present disclosure is directed towards systems and methods that leverage machine-learned models to decrease the rate at which abnormal sites are missed during a gastroenterological procedure. In particular, the system and methods of the present disclosure can use machine-learning techniques to determine the coverage rate achieved during a gastroenterological procedure. Measuring the coverage rate of the gastroenterological procedure can allow medical professionals to be alerted when the coverage output is deficient and thus allow an additional coverage to be achieved and as a result increase in the detection rate for abnormal sites (e.g., adenoma, polyp, lesion, tumor, etc.) during the gastroenterological procedure.

## Claims

### Claim 1

A computer-implemented method for determining coverage in gastroenterological procedures, the method comprising:
obtaining, by one or more computing devices, a plurality of images captured by an endoscopic device during a gastroenterological procedure for a patient, wherein the plurality of images depict respective portions of an anatomical structure viewed by the endoscopic device;
processing, by the one or more computing devices using a machine-learned depth estimation model, the plurality of images to obtain a plurality of depth maps respectively for the plurality of images, wherein the depth map obtained for each image describes one or more depths of the respective portions of the anatomical structure from the endoscopic device; and
determining, by the one or more computing devices using a machine-learned coverage estimation model, a coverage output of the anatomical structure based on the plurality of depth maps, wherein the coverage output indicates an amount of the anatomical structure which has been depicted by the plurality of images.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating, by the one or more computing devices, a visual presentation of the coverage output of the anatomical structure; and
providing, by the one or more computing devices, the visual presentation of the coverage output of the anatomical structure for use in completion of the gastroenterological procedure or diagnosis of the patient.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein providing, by the one or more computing devices, the visual presentation of the coverage output of the anatomical structure is performed in real-time during the gastroenterological procedure.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the coverage output indicates a percentage of potentially visible portions of the anatomical structure which were depicted by the plurality of images.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the machine-learned depth estimation model has been trained in an unsupervised training scheme using a view synthesis loss function that evaluates a reconstructed image that was reconstructed based on a depth map generated by the machine-learned depth estimation model.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the machine-learned coverage estimation model has been trained in a supervised training scheme using synthetic training data.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the machine-learned coverage estimation model comprises a first neural network that is configured to generate, for each depth map, a feature map and a second neural network that is configured to generate, from all feature maps, the coverage output for the gastroenterological procedure.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the first neural network and the second neural network have been separately trained.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, wherein the first neural network was trained to generate, from a depth map, a visibility metric indicative of a set of potentially visible points given a positioning of the endoscopic device.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the method is performed for each of a plurality of segments of a video of the gastroenterological procedure.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the anatomical structure comprises a colon and the gastroenterological procedure comprises a colonoscopy.

### Claim 12

A system for measuring coverage of a gastroenterological procedure, the system comprising:
a computing system comprising one or more processors and a non-transitory computer-readable memory;
wherein the non-transitory computer-readable memory stores instructions that, when executed by the processor, cause the computing system to perform operations, the operations comprising:
obtaining a plurality of images captured by an endoscopic device during a gastroenterological procedure for a patient, wherein the plurality of images depict respective portions of an anatomical structure viewed by the endoscopic device;
processing, using a machine-learned depth estimation model, the plurality of images to obtain a plurality of depth maps respectively for the plurality of images, wherein the depth map obtained for each image describes one or more depths of the respective portions of the anatomical structure from the endoscopic device; and
determining, using a machine-learned coverage estimation model, a coverage output of the anatomical structure based on the plurality of depth maps, wherein the coverage output indicates an amount of the anatomical structure which has been depicted by the plurality of images.

### Claim 13 (depends on 12)

The system of claim 12, the operations further comprising:
generating a visual presentation of the coverage output of the anatomical structure; and
providing the visual presentation of the coverage output of the anatomical structure for use in completion of the gastroenterological procedure or diagnosis of the patient.

### Claim 14 (depends on 12)

The system of claim 12, wherein providing the visual presentation of the coverage output of the anatomical structure is performed in real-time during the gastroenterological procedure.

### Claim 15 (depends on 12)

The system of any of claim 12, wherein the coverage output indicates a percentage of potentially visible portions of the anatomical structure which were depicted by the plurality of images.

### Claim 16 (depends on 12)

The system of any of claim 12, wherein the machine-learned depth estimation model has been trained in an unsupervised training scheme using a view synthesis loss function that evaluates a reconstructed image that was reconstructed based on a depth map generated by the machine-learned depth estimation model.

### Claim 17 (depends on 12)

The system of any of claim 12, wherein the machine-learned coverage estimation model has been trained in a supervised training scheme using synthetic training data.

### Claim 18 (depends on 12)

The system of any of claim 12, wherein the machine-learned coverage estimation model comprises a first neural network that is configured to generate, for each depth map, a feature map and a second neural network that is configured to generate, from all feature maps, the coverage output for the gastroenterological procedure.

### Claim 19

A non-transitory computer-readable medium storing instruction that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
obtaining a plurality of images captured by an endoscopic device during a gastroenterological procedure for a patient, wherein the plurality of images depict respective portions of an anatomical structure viewed by the endoscopic device;
processing, using a machine-learned depth estimation model, the plurality of images to obtain a plurality of depth maps respectively for the plurality of images, wherein the depth map obtained for each image describes one or more depths of the respective portions of the anatomical structure from the endoscopic device;
determining, using a machine-learned coverage estimation model, a coverage output of the anatomical structure based on the plurality of depth maps, wherein the coverage output indicates an amount of the anatomical structure which has been depicted by the plurality of images.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the operations further comprise:
generating a visual presentation of the coverage output of the anatomical structure; and
providing the visual presentation of the coverage output of the anatomical structure for use in completion of the gastroenterological procedure or diagnosis of the patient.
