# US 11,432,713 B2 · Endoscope system, endoscope image processing method, and computer readable recording medium for generating a three-dimensional shape image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,432,713 B2 - Document ID: US11432713B2 - Status: Granted - Kind: B2 - Application number: 16/672,736 - Filed: 2019-11-04 - Published: 2020-02-27 - Granted: 2022-09-06 - Earliest claimed: 2017-06-15 - Assignees: Olympus Corporation - Applicants: Olympus Corporation - Inventors: Shunya Akimoto - Web page: https://patentgrep.com/patent/US11432713B2 ## CPC - A61B 1/00194: {adapted for three-dimensional imaging} - A61B 1/00009: {of image signals during a use of endoscope} - A61B 1/05: characterised by the image sensor, e.g. camera, being in the distal end portion - G02B 23/24: Instruments {or systems} for viewing the inside of hollow bodies, e.g. fibrescopes - G06T 7/0012: {Biomedical image inspection} - G06T 7/579: from motion - G06T 7/73: using feature-based methods - G06T 2207/10016: Video; Image sequence - G06T 2207/10068: Endoscopic image - G06T 2207/30084: Kidney; Renal - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An endoscope system includes an endoscope configured to pick up an image of an inside of a subject to acquire an image, and a processor including hardware. The processor estimates a three-dimensional position of a target portion within the subject based on an image, sets for the three-dimensional position a reliability corresponding to a predetermined parameter related to the endoscope system determined when the image is acquired, and selects, when a plurality of three-dimensional positions for the target portion exist, a predetermined three-dimensional position among the plurality of three-dimensional positions depending on the reliability and generates a three-dimensional shape image. ## Claims ### Claim 1 An endoscope system comprising: an endoscope configured to pick up an image of an inside of a subject to acquire an image; and a processor including hardware, wherein the processor is configured to estimate a three-dimensional position of a target portion within the subject based on a predetermined image acquired by the endoscope, set, for the estimated three-dimensional position, a reliability corresponding to a predetermined parameter related to the endoscope system determined when the predetermined image is acquired, and select, when a plurality of three-dimensional positions for the target portion exist, a predetermined three-dimensional position among the plurality of three-dimensional positions depending on the set reliability and generate a three-dimensional shape image based on the selected predetermined three-dimensional position. ### Claim 2 (depends on 1) The endoscope system according to claim 1, wherein the processor selects the three-dimensional position for which the reliability is highest as the predetermined three-dimensional position. ### Claim 3 (depends on 2) The endoscope system according to claim 2, wherein the processor sets one of the estimated plurality of three-dimensional positions as a three-dimensional position of interest, draws a straight line between a position of a distal end of the endoscope when the image based on which the three-dimensional position of interest is estimated is acquired and the three-dimensional position of interest, to compare a reliability for another three-dimensional position on the straight line with a reliability for the three-dimensional position of interest, and selects the three-dimensional position of interest or the other three-dimensional position for which the reliability is higher. ### Claim 4 (depends on 3) The endoscope system according to claim 3, wherein the processor sets the estimated plurality of three-dimensional positions to the three-dimensional position of interest in descending order from the three-dimensional position for which the reliability is highest. ### Claim 5 (depends on 2) The endoscope system according to claim 2, wherein the processor sets one of the estimated plurality of three-dimensional positions to a three-dimensional position of interest, sets a vicinity range of a size corresponding to a reliability for the three-dimensional position of interest, to compare a reliability for another three-dimensional position within the vicinity range with the reliability for the three-dimensional position of interest, and selects the three-dimensional position of interest or the other three-dimensional position for which the reliability is higher. ### Claim 6 (depends on 5) The endoscope system according to claim 5, wherein the processor sets the estimated plurality of three-dimensional positions to the three-dimensional position of interest in descending order from the three-dimensional position for which the reliability is highest. ### Claim 7 (depends on 1) The endoscope system according to claim 1, wherein the processor selects, among the estimated plurality of three-dimensional positions, the three-dimensional position for which the reliability is judged to be higher than a threshold value as the predetermined three-dimensional position. ### Claim 8 (depends on 1) The endoscope system according to claim 1, wherein the processor generates the three-dimensional shape image that is made different in color for each portion depending on the reliability. ### Claim 9 (depends on 1) The endoscope system according to claim 1, wherein the processor estimates a distance between the endoscope and the subject based on the image, and estimates the three-dimensional position of the target portion based on the estimated distance and a position of the target portion in the image. ### Claim 10 (depends on 1) The endoscope system according to claim 1, wherein the processor acquires position information and posture information of the endoscope, and estimates the three-dimensional position of the target portion further using the position information and the posture information. ### Claim 11 (depends on 10) The endoscope system according to claim 10, comprising a sensor configured to measure a position and a posture of the endoscope, to acquire the position information and the posture information. ### Claim 12 (depends on 11) The endoscope system according to claim 11, wherein the predetermined parameter includes at least one of a luminance of the target portion appearing in the image, a light source aperture value for controlling a light amount of illumination light radiated onto the subject, a gain of an image pickup device configured to acquire the image, a position of the target portion appearing in the image relative to an optical center of the image, a hue of the target portion appearing in the image, a movement speed of a distal end of the endoscope, and a reliability for a measurement value by the sensor. ### Claim 13 (depends on 10) The endoscope system according to claim 10, wherein the processor estimates a position and a posture of the endoscope based on the image acquired by the endoscope, to acquire the position information and the posture information. ### Claim 14 An endoscope image processing method comprising: estimating a three-dimensional position of a target portion within a subject based on a predetermined image acquired by image pickup of an inside of the subject by an endoscope; setting for the estimated three-dimensional position a reliability corresponding to a predetermined parameter related to an endoscope system determined when the predetermined image is acquired; and selecting, when a plurality of three-dimensional positions for the target portion exist, a predetermined three-dimensional position among the plurality of three-dimensional positions depending on the set reliability and generating a three-dimensional shape image based on the selected predetermined three-dimensional position. ### Claim 15 A non-transitory computer readable recording medium storing an endoscope image processing program to be executed by a computer, the computer readable recording medium estimating, when a processor executes the endoscope image processing program, a three-dimensional position of a target portion within a subject based on a predetermined image acquired by image pickup of an inside of the subject by an endoscope; setting for the estimated three-dimensional position a reliability corresponding to a predetermined parameter related to the endo scope system determined when the predetermined image is acquired, and selecting, when a plurality of three-dimensional positions for the target portion exist, a predetermined three-dimensional position among the plurality of three-dimensional positions depending on the set reliability and generating a three-dimensional shape image based on the selected predetermined three-dimensional position.