# US 2024/0188798 A1 · Endoscope system, medical information processing apparatus, medical information processing method, medical information processing program, and recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0188798 A1
- Document ID: US20240188798A1
- Status: Published application
- Kind: A1
- Application number: 18/582,652
- Filed: 2024-02-21
- Published: 2024-06-13
- Earliest claimed: 2021-09-08
- Assignees: FUJIFILM Corporation
- Applicants: FUJIFILM Corporation
- Inventors: Yuya KIMURA; Yuma HORI; Tatsuya KOBAYASHI; Shigetoshi ISHIKAWA; Eiichi IMAMICHI
- Web page: https://patentgrep.com/patent/US20240188798A1

## CPC

- A61B 1/000096: {using artificial intelligence}
- A61B 1/045: Control thereof
- A61B 1/000094: {extracting biological structures}
- A61B 1/0002: {provided with data storages}
- A61B 1/0004: {for electronic operation}
- A61B 1/00042: {for mechanical operation}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/00055: {for alerting the user}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

According to the present disclosure, it is possible to smoothly proceed with an examination in which an audio input and audio recognition are performed on medical images. In the endoscope system, a processor acquires a plurality of medical images by causing an image sensor to image a subject in chronological order, accepts an input of an audio input trigger during capturing of the plurality of medical images, sets, in a case where the audio input trigger is input, an audio recognition dictionary according to the audio input trigger, performs, in a case where the audio recognition dictionary is set, audio recognition on audio input to the audio input device after the setting, using the set audio recognition dictionary, and displays item information indicating an item to be recognized using the audio recognition dictionary, and a result of audio recognition corresponding to the item information, on a display device.

## Claims

### Claim 1

An endoscope system comprising:
an audio input device;
an image sensor that images a subject; and
a processor,
wherein the processor is configured to:
acquire a plurality of medical images obtained by the image sensor imaging the subject in chronological order;
perform a plurality of image recognitions on the acquired plurality of medical images;
set, in a case where a result of any one of the plurality of image recognitions is obtained, an audio recognition dictionary according to the obtained result of any one of the plurality of image recognitions;
display, in a case where the audio recognition dictionary is set, item information indicating an item to be recognized using the audio recognition dictionary, on a display device;
perform audio recognition using the set audio recognition dictionary, in a case where audio input is performed for the item information displayed on the display device; and
display a result of the audio recognition in association with the item information, on the display device.

### Claim 2 (depends on 1)

The endoscope system according to claim 1,
wherein the processor performs image recognition for detecting a specific subject, as one of the plurality of image recognitions.

### Claim 3 (depends on 2)

The endoscope system according to claim 2,
wherein the specific subject is at least one of a treatment tool and a hemostat.

### Claim 4 (depends on 3)

The endoscope system according to claim 3,
wherein, in a case where a result of image recognition indicating detection of the treatment tool is obtained, the processor displays an item of treatment as the item information on the display device,
and/or
in a case where a result of image recognition indicating detection of the hemostat is obtained, the processor displays an item of hemostat, as the item information, on the display device.

### Claim 5 (depends on 2)

The endoscope system according to claim 2,
wherein the processor further performs image recognition for discrimination of a lesion detected from the plurality of medical images, as one of the plurality of image recognitions.

### Claim 6 (depends on 5)

The endoscope system according to claim 5,
wherein, in a case where a result of the discrimination is obtained, the processor displays an item of diagnosis and findings, as the item information, on the display device.

### Claim 7 (depends on 1)

The endoscope system according to claim 1,
wherein, in the audio recognition, the processor recognizes only registered words that are registered in the set audio recognition dictionary, and displays the result of the audio recognition for the registered words on the display device.

### Claim 8 (depends on 1)

The endoscope system according to claim 1,
wherein, in the audio recognition, the processor recognizes registered words that are registered in the set audio recognition dictionary and specific words, and displays the result of the audio recognition for the registered words among the recognized words on the display device.

### Claim 9 (depends on 1)

The endoscope system according to claim 1,
wherein the processor performs the plurality of image recognitions, using an image recognizer generated by machine learning.

### Claim 10 (depends on 1)

The endoscope system according to claim 1,
wherein the processor displays information indicating that the audio recognition dictionary is set, on the display device.

### Claim 11 (depends on 1)

The endoscope system according to claim 1,
wherein the processor displays type information indicating a type of the set audio recognition dictionary, on the display device.

### Claim 12 (depends on 1)

The endoscope system according to claim 1,
wherein the processor displays the item information and the result of the audio recognition in chronological order.

### Claim 13 (depends on 1)

The endoscope system according to claim 1,
wherein the processor displays the item information and the result of the audio recognition on a same display screen as the plurality of medical images.

### Claim 14 (depends on 1)

The endoscope system according to claim 1,
wherein the processor
accepts confirmation information indicating confirmation of the audio recognition, and ends, in a case where the confirmation information is accepted, display of the result of the audio recognition and the item information.

### Claim 15 (depends on 1)

The endoscope system according to claim 1,
wherein the processor displays the item information and the result of the audio recognition during a display period after the setting, and ends the display in a case where the display period has elapsed.

### Claim 16 (depends on 15)

The endoscope system according to claim 15,
wherein the processor displays the item information and the result of the audio recognition during a period in which the audio recognition dictionary is set, as the display period, and ends the display of the item information and the result of the audio recognition in a case where the display period ends.

### Claim 17 (depends on 15)

The endoscope system according to claim 15,
wherein the processor displays the item information and the result of the audio recognition during a period with a length according to the obtained result of any one of the plurality of image recognitions, as the display period, and ends the display of the item information and the result of the audio recognition in a case where the display period ends.

### Claim 18 (depends on 15)

The endoscope system according to claim 15,
wherein the processor ends the display of the item information and the result of the audio recognition in a case where a state in which a specific subject is recognized in the plurality of medical images ends.

### Claim 19 (depends on 15)

The endoscope system according to claim 15,
wherein the processor displays a remaining time of the display period on a screen of the display device.

### Claim 20 (depends on 1)

The endoscope system according to claim 1,
wherein the processor
displays a candidate for recognition in the audio recognition on the display device, and
confirms the result of the audio recognition on the basis of a selection operation of a user according to the display of the candidate.

### Claim 21 (depends on 20)

The endoscope system according to claim 20,
wherein the processor accepts the selection operation via an operation device different from the audio input device.

### Claim 22 (depends on 1)

The endoscope system according to claim 1,
wherein the processor records the plurality of medical images, the item information, and the result of the audio recognition in a recording device such that the plurality of medical images are associated with the item information, and the result of the audio recognition.

### Claim 23

A medical information processing apparatus comprising
a processor configured to:
acquire a plurality of medical images obtained by an image sensor imaging a subject in chronological order;
perform a plurality of image recognitions on the acquired plurality of medical images;
set, in a case where a result of any one of the plurality of image recognitions is obtained, an audio recognition dictionary according to the obtained result of any one of the plurality of image recognitions;
display, in a case where the audio recognition dictionary is set, item information indicating an item to be recognized using the audio recognition dictionary, on a display device;
perform audio recognition using the set audio recognition dictionary, in a case where audio input is performed for the item information displayed on the display device; and
display a result of the audio recognition in association with the item information, on the display device.

### Claim 24

A medical information processing method executed by an endoscope system including an audio input device, an image sensor that images a subject, and a processor, the medical information processing method comprising:
acquiring, by the processor, a plurality of medical images obtained by the image sensor imaging the subject in chronological order;
performing, by the processor, a plurality of image recognitions on the acquired plurality of medical images;
setting, by the processor, in a case where a result of any one of the plurality of image recognitions is obtained, an audio recognition dictionary according to the obtained result of any one of the plurality of image recognitions;
displaying, by the processor, in a case where the audio recognition dictionary is set, item information indicating an item to be recognized using the audio recognition dictionary, on a display device;
performing, by the processor, audio recognition using the set audio recognition dictionary, in a case where audio input is performed for the item information displayed on the display device; and
displaying, by the processor, a result of the audio recognition in association with the item information, on the display device.

### Claim 25

A non-transitory, tangible recording medium which records thereon a computer-readable code of a program for causing a processor of an endoscope system including an audio input device and an image sensor that images a subject, to implement functions comprising:
acquiring a plurality of medical images obtained by the image sensor imaging the subject in chronological order;
performing a plurality of image recognitions on the acquired plurality of medical images;
setting, in a case where a result of any one of the plurality of image recognitions is obtained, an audio recognition dictionary according to the obtained result of any one of the plurality of image recognitions;
displaying, in a case where the audio recognition dictionary is set, item information indicating an item to be recognized using the audio recognition dictionary, on a display device;
performing, audio recognition using the set audio recognition dictionary, in a case where audio input is performed for the item information displayed on the display device; and
displaying, a result of the audio recognition in association with the item information, on the display device.
