# US 2023/0410491 A1 · Multi-view medical activity recognition systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410491 A1
- Document ID: US20230410491A1
- Status: Published application
- Kind: A1
- Application number: 18/035,993
- Filed: 2021-11-12
- Published: 2023-12-21
- Earliest claimed: 2020-11-13
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: Omid Mohareri; Adam T. Schmidt; Aidean Sharghi Karganroodi
- Web page: https://patentgrep.com/patent/US20230410491A1

## CPC

- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 10/993: {Evaluation of the quality of the acquired pattern}
- G06F 18/24133: {Distances to prototypes}
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/73: using feature-based methods
- G06V 10/12: Details of acquisition arrangements; Constructional details thereof
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/82: using neural networks
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image}
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30168: Image quality inspection
- G06T 2207/30244: Camera pose
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Multi-view medical activity recognition systems and methods are described herein. In certain illustrative examples, a system accesses a plurality of data streams representing imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints. The system temporally aligns the plurality of data streams and determines, using a viewpoint agnostic machine learning model and based on the plurality of data streams, an activity within the scene.

## Claims

### Claim 1

A system comprising:
a memory storing instructions;
a processor communicatively coupled to the memory and configured to execute the instructions to:
access a plurality of data streams representing imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the plurality of sensors including a dynamic sensor capturing the imagery from a dynamic viewpoint that changes during the medical session;
temporally align the plurality of data streams; and
determine, using a viewpoint agnostic machine learning model and based on the plurality of data streams, an activity within the scene.

### Claim 2 (depends on 1)

The system of claim 1, wherein:
the machine learning model is configured to generate fused data based on the plurality of data streams; and
the determining the activity within the scene is based on the fused data.

### Claim 3 (depends on 2)

The system of claim 2, wherein:
the plurality of data streams comprises a first data stream and a second data stream;
the machine learning model is further configured to:
determine, based on the first data stream, a first classification of the activity within the scene, and
determine, based on the second data stream, a second classification of the activity within the scene; and
the generating the fused data comprises combining the first classification and the second classification using a weighting determined based on the first data stream, the second data stream, and the activity within the scene.

### Claim 4 (depends on 2)

The system of claim 2, wherein:
the plurality of data streams comprises a first data stream and a second data stream; and
the generating the fused data comprises:
determining, based on the first data stream and the second data stream, a global classification of the activity within the scene,
determining, based on the first data stream and the global classification, a first classification of the activity within the scene,
determining, based on the second data stream and the global classification, a second classification of the activity within the scene, and
combining the first classification, the second classification, and the global classification using a weighting determined based on the first data stream, the second data stream, and the activity within the scene.

### Claim 5 (depends on 4)

The system of claim 4, wherein the determining the global classification comprises combining, for points in time, respective temporally aligned data from the first data stream and the second data stream corresponding to the points in time using a weighting determined based on the first data stream, the second data stream, and the activity within the scene.

### Claim 6 (depends on 4)

The system of claim 4, wherein the determining the global classification comprises:
extracting first features from the data of the first data stream;
extracting second features from the data of the second data stream; and
combining the first features and the second features using a weighting determined based on the first data stream, the second data stream, and the activity within the scene.

### Claim 7 (depends on 1)

The system of claim 1, wherein the determining the activity within the scene is performed during the activity within the scene.

### Claim 8 (depends on 1)

The system of claim 1, wherein the plurality of data streams further comprises a data stream representing data captured by a non-imaging sensor.

### Claim 9 (depends on 1)

The system of claim 1, wherein the viewpoint agnostic model is agnostic to a number of the plurality of sensors.

### Claim 10 (depends on 1)

The system of claim 1, wherein the viewpoint agnostic model is agnostic to positions of the plurality of sensors.

### Claim 11

A method comprising:
accessing, by a processor, a plurality of data streams representing imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the plurality of sensors including a dynamic sensor capturing the imagery from a dynamic viewpoint that changes during the medical session;
temporally aligning, by the processor, the plurality of data streams; and
determining, by the processor, using a viewpoint agnostic machine learning model and based on the plurality of data streams, an activity within the scene.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
the machine learning model is configured to generate fused data based on the plurality of data streams; and
the determining the activity within the scene is based on the fused data.

### Claim 13 (depends on 12)

The method of claim 12, wherein:
the plurality of data streams comprises a first data stream and a second data stream;
the machine learning model is further configured to:
determine, based on the first data stream, a first classification of the activity within the scene, and
determine, based on the second data stream, a second classification of the activity within the scene; and
the generating the fused data comprises combining the first classification and the second classification using a weighting determined based on the first data stream, the second data stream, and the activity within the scene.

### Claim 14 (depends on 12)

The method of claim 12, wherein:
the plurality of data streams comprises a first data stream and a second data stream; and
the generating the fused data comprises:
determining, based on the first data stream and the second data stream, a global classification of the activity within the scene,
determining, based on the first data stream and the global classification, a first classification of the activity within the scene,
determining, based on the second data stream and the global classification, a second classification of the activity within the scene, and
combining the first classification, the second classification, and the global classification using a weighting determined based on the first data stream, the second data stream, and the activity within the scene.

### Claim 15 (depends on 14)

The method of claim 14, wherein the determining the global classification comprises combining, for points in time, respective temporally aligned data from the first data stream and the second data stream corresponding to the points in time using a weighting determined based on the first data stream, the second data stream, and the activity within the scene.

### Claim 16 (depends on 14)

The method of claim 14, wherein the determining the global classification comprises:
extracting first features from the data of the first data stream;
extracting second features from the data of the second data stream; and
combining the first features and the second features using a weighting determined based on the first data stream, the second data stream, and the activity within the scene.

### Claim 17 (depends on 11)

The method of claim 11, wherein the determining the activity within the scene is performed during the activity within the scene.

### Claim 18 (depends on 11)

The method of claim 11, wherein the plurality of data streams further comprises a data stream representing data captured by a non-imaging sensor.

### Claim 19

A non-transitory computer-readable medium storing instructions executable by a processor to:
Access a plurality of data streams representing imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the plurality of sensors including a dynamic sensor capturing the imagery from a dynamic viewpoint that changes during the medical session;
temporally align the plurality of data streams; and
determine, using a viewpoint agnostic machine learning model and based on the plurality of data streams, an activity within the scene.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein:
the machine learning model is configured to generate fused data based on the plurality of data streams; and
the determining the activity within the scene is based on the fused data.

### Claim at position 21

21-26. (canceled)
