# US 2023/0410499 A1 · Visibility metrics in multi-view medical activity recognition systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410499 A1
- Document ID: US20230410499A1
- Status: Published application
- Kind: A1
- Application number: 18/035,989
- Filed: 2021-11-12
- Published: 2023-12-21
- Earliest claimed: 2020-11-13
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: Omid Mohareri; Adam T. Schmidt; Aidean Sharghi Karganroodi
- Web page: https://patentgrep.com/patent/US20230410499A1

## CPC

- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 10/993: {Evaluation of the quality of the acquired pattern}
- G06F 18/24133: {Distances to prototypes}
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/73: using feature-based methods
- G06V 10/12: Details of acquisition arrangements; Constructional details thereof
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/82: using neural networks
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image}
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30168: Image quality inspection
- G06T 2207/30244: Camera pose
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Visibility metrics in multi-view medical activity recognition systems and methods are described herein. In certain illustrative examples, a system access imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the imagery including first imagery captured by a first sensor of the plurality of sensors from a first viewpoint of the plurality of viewpoints. The system determines, during the medical session and based on the first imagery, a value of an activity visibility metric for the first sensor. The system facilitates, based on the value of the activity visibility metric, adjusting the first viewpoint of the first sensor.

## Claims

### Claim at position 1

1-32. (canceled)

### Claim 33

A system comprising:
a memory storing instructions;
a processor communicatively coupled to the memory and configured to execute the instructions to:
access imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the imagery including first imagery captured by a first sensor of the plurality of sensors from a first viewpoint of the plurality of viewpoints;
determine, during the medical session and based on the first imagery, a value of an activity visibility metric for the first sensor; and
facilitate, based on the value of the activity visibility metric, adjusting the first viewpoint of the first sensor.

### Claim 34 (depends on 33)

The system of claim 33, wherein the facilitating adjusting the first viewpoint of the first sensor comprises providing an output to a robotic system to instruct the robotic system to change a pose of the first sensor.

### Claim 35 (depends on 33)

The system of claim 33, wherein the facilitating adjusting the first viewpoint of the first sensor comprises providing an output to a user to instruct the user to change a pose of the first sensor.

### Claim 36 (depends on 33)

The system of claim 33, wherein:
the instructions comprise a machine learning model trained based on training imagery labeled with an activity of scenes captured in the training imagery; and
the determining the value of the activity visibility metric for the first sensor comprises using the machine learning model.

### Claim 37 (depends on 33)

The system of claim 33, wherein the processor is further configured to execute the instructions to:
access additional imagery of the scene of the medical session captured by the plurality of sensors from another plurality of viewpoints, the additional imagery including second imagery captured by the first sensor from a second viewpoint different from the first viewpoint; and
determine, based on the additional imagery, an additional value of the activity visibility metric that is higher than the value of the activity visibility metric.

### Claim 38 (depends on 33)

The system of claim 33, wherein:
the imagery of the scene includes:
second imagery captured by a second sensor of the plurality of sensors from a second viewpoint of the plurality of viewpoints, and
third imagery captured by a third sensor of the plurality of sensors from a third viewpoint of the plurality of viewpoints; and
the processor is further configured to execute the instructions to:
determine that the value of the activity visibility metric for the first sensor is below a threshold value of the activity visibility metric, and
use, based on the determining that the value of the activity visibility metric for the first sensor is below the threshold value of the activity visibility metric, a generative model to produce generated imagery based on the second imagery and the third imagery.

### Claim 39 (depends on 38)

The system of claim 38, wherein the generated imagery comprises imagery generated based on the first viewpoint, the generated imagery having a generated value of the activity visibility metric that is higher than the value of the activity visibility metric for the first sensor.

### Claim 40 (depends on 38)

The system of claim 38, wherein:
the generated imagery comprises imagery generated based on a fourth viewpoint, the generated imagery having a generated value of the activity visibility metric that is higher than the value of the activity visibility metric for the first sensor; and
the facilitating adjusting the first viewpoint of the first sensor comprises providing an output comprising an instruction to change a pose of the first sensor to capture additional imagery of the scene from the fourth viewpoint.

### Claim 41 (depends on 38)

The system of claim 38, wherein:
the processor is further configured to execute the instructions to:
determine, based on the second imagery, a value of the activity visibility metric for the second sensor,
determine, based on the third imagery, a value of the activity visibility metric for the third sensor; and
the using the generative model to produce generated imagery based on the second imagery and the third imagery is further based on the values of the activity visibility metric for the second sensor and the third sensor being at least the threshold value of the activity visibility metric.

### Claim 42 (depends on 33)

The system of claim 33, wherein:
the imagery of the scene includes second imagery captured by a second sensor of the plurality of sensors from a second viewpoint of the plurality of viewpoints;
the processor is further configured to execute the instructions to determine, based on the first imagery and the second imagery, an overall value of the activity visibility metric for the plurality of sensors; and
the facilitating adjusting the first viewpoint of the first sensor comprises adjusting the first viewpoint to improve the overall value of the activity visibility metric.

### Claim 43 (depends on 42)

The system of claim 42, wherein the facilitating adjusting the first viewpoint results in a lower value of the activity visibility metric for the first sensor and a higher overall value of the activity visibility metric for the plurality of sensors.

### Claim 44 (depends on 33)

The system of claim 33, wherein the value of the activity visibility metric represents a rating of how visible an activity of the scene is in the first imagery.

### Claim 45

A system comprising:
a memory storing instructions;
a processor communicatively coupled to the memory and configured to execute the instructions to:
access imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the imagery including first imagery captured by a first sensor of the plurality of sensors from a first viewpoint of the plurality of viewpoints;
determine, based on the first imagery, a first classification of an activity of the scene;
determine, during the medical session and based on the first imagery, a value of an activity visibility metric for the first sensor;
determine that the value of the activity visibility metric for the first sensor is below a threshold value of the activity visibility metric; and
lower, based on the determining that the value of the activity visibility metric for the first sensor is below the threshold value of the activity visibility metric, a weighting of the first classification of the activity of the scene for determining an overall classification of the activity of the scene based on the imagery of the scene.

### Claim 46 (depends on 45)

The system of claim 45, wherein the processor is further configured to execute the instructions to facilitate, based on the determining that the value of the activity visibility metric for the first sensor is below the threshold value of the activity visibility metric, adjusting the first viewpoint of the first sensor.

### Claim 47

A method comprising:
accessing, by a processor, imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the imagery including first imagery captured by a first sensor of the plurality of sensors from a first viewpoint of the plurality of viewpoints;
determining, by the processor, during the medical session and based on the first imagery, a value of an activity visibility metric for the first sensor; and
facilitating, by the processor, based on the value of the activity visibility metric, adjusting the first viewpoint of the first sensor.

### Claim 48 (depends on 47)

The method of claim 47, wherein the facilitating adjusting the first viewpoint of the first sensor comprises providing an output to a robotic system to instruct the robotic system to change a pose of the first sensor.

### Claim 49 (depends on 47)

The method of claim 47, wherein the facilitating adjusting the first viewpoint of the first sensor comprises providing an output to a user to instruct the user to change a pose of the first sensor.

### Claim 50 (depends on 47)

The method of claim 47, wherein the determining the value of the activity visibility metric for the first sensor comprises using a machine learning model trained based on training imagery labeled with an activity of scenes captured in the training imagery.

### Claim 51 (depends on 47)

The method of claim 47, further comprising:
accessing, by the processor, additional imagery of the scene of the medical session captured by the plurality of sensors from another plurality of viewpoints, the additional imagery including second imagery captured by the first sensor from a second viewpoint different from the first viewpoint; and
determining, by the processor, based on the additional imagery, an additional value of the activity visibility metric that is higher than the value of the activity visibility metric.

### Claim 52 (depends on 47)

The method of claim 47, wherein:
the imagery of the scene includes:
second imagery captured by a second sensor of the plurality of sensors from a second viewpoint of the plurality of viewpoints, and
third imagery captured by a third sensor of the plurality of sensors from a third viewpoint of the plurality of viewpoints; and
the method further comprises:
determining, by the processor, that the value of the activity visibility metric for the first sensor is below a threshold value of the activity visibility metric, and
using, by the processor, based on the determining that the value of the activity visibility metric for the first sensor is below the threshold value of the activity visibility metric, a generative model to produce generated imagery based on the second imagery and the third imagery.
