# US 11,992,177 B2 · Image processing device for endoscope, image processing method for endoscope, and recording medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,992,177 B2 - Document ID: US11992177B2 - Status: Granted - Kind: B2 - Application number: 17/238,865 - Filed: 2021-04-23 - Published: 2021-08-19 - Granted: 2024-05-28 - Earliest claimed: 2018-10-26 - Assignees: Olympus Corporation - Applicants: Olympus Corporation - Inventors: Mitsutaka Kimura; Yamato Kanda; Makoto Kitamura; Katsuyoshi Taniguchi - Web page: https://patentgrep.com/patent/US11992177B2 ## CPC - A61B 1/000094: {extracting biological structures} - A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled} - A61B 1/00055: {for alerting the user} - A61B 1/045: Control thereof - A61B 1/05: characterised by the image sensor, e.g. camera, being in the distal end portion - G06T 7/0012: {Biomedical image inspection} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 2207/10068: Endoscopic image - G06T 2207/30096: Tumor; Lesion - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image processing device for an endoscope includes a processor. The processor receives, as input, a generated image acquired by performing predetermined processing on an image pickup signal acquired by an endoscope, analyzes a withdrawing speed of the endoscope based on at least one of the generated image and information relating to an operation state of the endoscope, detects a lesion part from the generated image, performs a diagnosis support action on the lesion part, decides a factor affecting the diagnosis support action based on an analysis result of the withdrawing speed, and outputs the factor. ## Claims ### Claim 1 An image processing device for an endoscope, comprising: a processor configured to: receive a generated image generated from an image of an object captured by an endoscope; calculate a withdrawing speed of the endoscope based on at least one of the generated image and sensor information relating to an operation state of the endoscope; detect a lesion part that is an observation target from the generated image; perform a diagnosis support action of adding support information to the generated image in which the lesion part was detected; determine whether the withdrawing speed of the endoscope is within a predetermined range; and in response to determining that the withdrawing speed of the endoscope is not within the predetermined range, determine that the lesion part cannot be detected or is erroneously detected, and control a display to display a warning content that the diagnosis support function is not performed normally. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to: receive system information of system equipment including the endoscope; perform two or more of: a first determination of whether the lesion part cannot be detected or is erroneously detected based on the system information; a second determination of whether the lesion part cannot be detected or is erroneously detected based on an image pickup state of the generated image; and a third determination of whether the lesion part cannot be detected or is erroneously detected based on an operation state of the endoscope; determine that one of a result of the first determination, a result of the second determination and a result of the third determination has a highest priority; and in response to determining that the one of the result of the first determination, the result of the second determination and the one of the result of the third determination, determined to having the highest priority, is that the lesion part cannot be detected or is erroneously detected, control the display to display the warning content regardless of the other of the result of the first determination, the result of the second determination and the result of the third determination. ### Claim 3 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to; determine whether the lesion part cannot be detected or is erroneously detected based on an image pickup state of the generated image; determine whether the lesion part cannot be detected or is erroneously detected based on an operation state of the endoscope; and in response to determining that the lesion part cannot be detected or is erroneously detected based on the image pickup state, control the display to display the warning content regardless of a result of a determination of whether the lesion part cannot be detected or is erroneously detected based on the operation state of the endoscope. ### Claim 4 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to; receive system information of system equipment including the endoscope; determine whether the lesion part cannot be detected or is erroneously detected based on the system information; determine whether the lesion part cannot be detected or is erroneously detected based on an operation state of the endoscope; and in response to determining that the lesion part cannot be detected or is erroneously detected based on the system information, control the display to display the warning content regardless of a result of a determination of whether the lesion part cannot be detected or is erroneously detected based on the operation state of the endoscope. ### Claim 5 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to; receive system information of system equipment including the endoscope; determine whether the lesion part cannot be detected or is erroneously detected based on the system information; determine whether the lesion part cannot be detected or is erroneously detected based on an image pickup state of the generated image; and in response to determining that the lesion part cannot be detected or is erroneously detected based on the system information, control the display to display the warning content regardless of a result of a determination of whether the lesion part cannot be detected or is erroneously detected based on the image pickup state of the generated image. ### Claim 6 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to; receive system information of system equipment including the endoscope; determine whether the lesion part cannot be detected or is erroneously detected based on the system information; and in response to determining that the lesion part cannot be detected or is erroneously detected based on the system information, control the display to display the warning content. ### Claim 7 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to; receive a set parameter of system equipment including the endoscope; determine whether the lesion part cannot be detected or is erroneously detected based on the set parameter of the system equipment; and in response to determining that the lesion part cannot or is erroneously detected based on the set parameter, control the display to display the warning content. ### Claim 8 (depends on 7) The image processing device according to claim 7, wherein the set parameter of the system equipment comprises a value of a parameter of image highlight processing performed on the generated image or an image pickup signal from which the generated image is generated. ### Claim 9 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to; determine whether an algorithm which is set to execute the diagnosis support action is compatible with system equipment including the endoscope; determine whether the lesion part cannot be detected or is erroneously detected based on a result of the determination of whether the algorithm which is set to execute the diagnosis support action is compatible with the system equipment including the endoscope; and in response to determining that the lesion part cannot be detected or is erroneous detected based on the result of the determination of whether the algorithm which is set to execute the diagnosis support action is compatible with the system equipment including the endoscope, control the display to display the warning content. ### Claim 10 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to; determine, based on the generated image, whether the lesion part cannot be detected or is erroneously detected due to an image pickup state of the endoscope; and in response to determining that the lesion part cannot be detected or is erroneously detected due to the image pickup state of the endoscope, control the display to display the warning content. ### Claim 11 (depends on 10) The image processing device according to claim 10, wherein the processor is configured to: in determining whether the lesion part cannot be detected or is erroneously detected due to the image pickup state of the endoscope, determine whether an image pickup target in the generated image is not focused; and in response to determining that the image pickup target in the generated image is not focused, determine that the lesion part cannot be detected or is erroneously detected due to the image pickup state of the endoscope. ### Claim 12 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to: determine whether the lesion part cannot be detected or is erroneously detected due to a state of the object shown in the generated image; and in response to determining that the lesion part cannot be detected or is erroneously detected due to the state of the object shown in the generated image, control the display to display the warning content. ### Claim 13 (depends on 12) The image processing device according to claim 12, wherein the processor is configured to; in determining whether the lesion part cannot be detected or is erroneously detected due to the state of the object shown in the generated image, determine whether an image portion including one or more of debris, bleeding and a treatment scar is present in the generated image; and in response to determining that the image portion including one or more of the debris, the bleeding and the treatment scar is present in the generated image, determine that the lesion part cannot be detected or is erroneously detected due to the state of the object shown in the generated image. ### Claim 14 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to; analyze at least one of the generated image and the information relating to the operation state of the endoscope to determine whether the lesion part cannot be detected or is erroneously detected due to one or more factors thereby causing the diagnosis support function to not perform normally, wherein the one or more factors includes the withdrawing speed of the endoscope being not within the predetermined range: determine a solution method for removing the one or more factors; and control the display to display the solution method. ### Claim 15 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to: determine reliability of the diagnosis support action; and control the display to display the reliability of the diagnosis support action determined. ### Claim 16 (depends on 1) The image processing device according to claim 1, wherein the processor is configured to: determine whether the lesion part cannot be detected or is erroneously detected due to one or more external factors thereby causing the diagnosis support function to not perform normally, wherein the one or more external factors includes determining that the withdrawing speed of the endoscope is not within the predetermined range; and in response to determining that the lesion part cannot be detected or is erroneously detected due to the one or more external factors, stop performing the diagnosis support action. ### Claim 17 An image processing method for an endoscope, the image processing method comprising: receiving a generated image generated from an image of an object captured by an endoscope; calculating a withdrawing speed of the endoscope based on at least one of the generated image and sensor information relating to an operation state of the endoscope; detecting a lesion part that is an observation target from the generated image; performing a diagnosis support action of adding support information to the generated image in which the lesion part was detected; determining the withdrawing speed of the endoscope is within a predetermined range; and in response to determining that the withdrawing speed of the endoscope is not within the predetermined range, determining that the lesion part cannot be detected or is erroneously detected and controlling a display to display a warning content that the diagnosis support function is not performed normally. ### Claim 18 (depends on 17) The image processing method according to claim 17, the image processing method further comprising: receiving system information of system equipment including the endoscope; performing two or more of: a first determination of whether the lesion part cannot be detected or is erroneously detected based on the system information; a second determination of whether the lesion part cannot be detected or is erroneously detected based on an image pickup state of the generated image; a third determination of whether the lesion part cannot be detected or is erroneously detected based on an operation state of the endoscope; determining that one of a result of the first determination, a result of the second determination and a result of the third determination has a highest priority; and in response to determining that the one of the result of the first determination, the result of the second determination and the one of the result of the third determination, determined to having the highest priority, is that the lesion part cannot be detected or is erroneously detected, control the display to display the warning content regardless of the other of the result of the first determination, the result of the second determination and the result of the third determination. ### Claim 19 (depends on 17) The image processing method according to claim 17, the image processing method further comprising: receiving a set parameter of system equipment including the endoscope; determining whether the lesion part cannot be detected or is erroneously detected based on the set parameter of the system equipment; and in response to determining that the lesion part cannot or is erroneously detected based on the set parameter, control the display to display the warning content. ### Claim 20 (depends on 17) The image processing method according to claim 17, the image processing method further comprising: determining reliability of the diagnosis support action; and controlling the display to display the reliability of the diagnosis support action determined. ### Claim 21 A non-transitory computer-readable recording medium storing a computer program that causes a computer to at least: receive a generated image generated from an image of an object captured by an endoscope; calculate a withdrawing speed of the endoscope based on at least one of the generated image and sensor information relating to an operation state of the endoscope; detect a lesion part that is an observation target from the generated image; perform a diagnosis support action of adding support information to the generated image in which the lesion part was detected; and determine whether the withdrawing speed of the endoscope is within a predetermined range; and in response to determining that the withdrawing speed of the endoscope is not within the predetermined range, determine that the lesion part cannot be detected or is erroneously detected, and control a display to display a warning content that the diagnosis support function is not performed normally. ### Claim 22 (depends on 21) The non-transitory computer-readable recording medium according to claim 21, wherein the computer program causes the computer to: receive system information of system equipment including the endoscope; perform two or more of: a first determination of whether the lesion part cannot be detected or is erroneously detected based on the system information; a second determination of whether the lesion part cannot be detected or is erroneously detected based on an image pickup state of the generated image; and a third determination of whether the lesion part cannot be detected or is erroneously detected based on an operation state of the endoscope; determine that one of a result of the first determination, a result of the second determination and a result of the third determination has a highest priority; and in response to determining that the one of the result of the first determination, the result of the second determination and the one of the result of the third determination, determined to having the highest priority, is that the lesion part cannot be detected or is erroneously detected, control the display to display the warning content regardless of the other of the result of the first determination, the result of the second determination and the result of the third determination. ### Claim 23 (depends on 21) The non-transitory computer-readable recording medium according to claim 21, wherein the computer program causes the computer to: receive a set parameter of system equipment including the endoscope; determine whether the lesion part cannot be detected or is erroneously detected based on the set parameter of the system equipment; and in response to determining that the lesion part cannot or is erroneously detected based on the set parameter, control the display to display the warning content. ### Claim 24 (depends on 21) The non-transitory computer-readable recording medium according to claim 21, wherein the computer program causes the computer to: determine reliability of the diagnosis support action; and control the display to display the reliability of the diagnosis support action determined.