# US 12,112,484 B2 · Method for generating learning model and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,484 B2 - Document ID: US12112484B2 - Status: Granted - Kind: B2 - Application number: 17/627,327 - Filed: 2020-11-18 - Published: 2022-09-01 - Granted: 2024-10-08 - Earliest claimed: 2019-12-05 - Assignees: Hoya Corporation - Applicants: Hoya Corporation - Inventors: Chihiro Kambe; Kohei Iketani - Web page: https://patentgrep.com/patent/US12112484B2 ## CPC - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS - A61B 1/00: Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes (examination of body cavities or body tracts using ultrasonic, sonic or infrasonic waves A61B8/12; endoscopic instruments for taking cell samples or for biopsy A61B10/04); Illuminating arrangements therefor (for the eyes A61B3/00) - A61B 1/045: Control thereof - G02B 23/24: Instruments {or systems} for viewing the inside of hollow bodies, e.g. fibrescopes - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06T 7/0012: {Biomedical image inspection} - G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture - G16H 30/40: for processing medical images, e.g. editing - G16H 40/63: for local operation - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10068: Endoscopic image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract There is provided a method for generating a learning model, the method including: acquiring an endoscopic image captured by an endoscope and manipulation information regarding a manipulation of an endoscope operator in each stage of operation of the endoscope by the endoscope operator operating the endoscope; and generating a learning model learned so as to output the manipulation information of a next stage in a case where the endoscopic image and the manipulation information are input, based on training data including the acquired endoscopic image and manipulation information, and the manipulation information of the next stage. ## Claims ### Claim 1 A method for generating a learning model, the method comprising: acquiring an endoscopic image captured by an endoscope and manipulation information regarding a manipulation of an endoscope operator in each stage of operation of the endoscope by the endoscope operator operating the endoscope; generating a learning model learned so as to output the manipulation information of a next stage in a case where the endoscopic image and the manipulation information are input, based on training data including the acquired endoscopic image and manipulation information, and the manipulation information of the next stage; specifying the manipulation level of the endoscope operator with respect to the manipulation of the endoscope; displaying the manipulation information of the next stage, which is output from the learning model, in a case of a first manipulation level according to the specified manipulation level; and in a case of a second manipulation level higher than the first manipulation level, not displaying the manipulation information of the next stage, which is output from the learning model, when predetermined conditions are satisfied, and displaying the manipulation information of the next stage, which is output from the learning model, when the predetermined conditions are not satisfied. ### Claim 2 (depends on 1) The method for generating a learning model according to claim 1, further comprising: acquiring the manipulation information indicating state data of both arms and both hands of the endoscope operator by using a 3D laser sensor; and generating the learning model learned so as to output the manipulation information indicating the state data of both arms and both hands of the next stage in a case where the endoscopic image and the manipulation information indicating the state data of both arms and both hands are input to the learning model. ### Claim 3 (depends on 1) The method for generating a learning model according to claim 1, further comprising: acquiring the manipulation information indicating operation data of a distal end of the endoscope; and generating the learning model learned so as to output the manipulation information indicating the operation data of the distal end of the endoscope of the next stage in a case where the endoscopic image and the manipulation information indicating the operation data of the distal end of the endoscope are input to the learning model. ### Claim 4 (depends on 1) The method for generating a learning model according to claim 1, further comprising: generating the learning model learned so as to output an image based on the manipulation information indicating the state data of both arms and both hands of the next stage in a case where the endoscopic image and the manipulation information indicating the state data of both arms and both hands are input to the learning model. ### Claim 5 (depends on 1) The method for generating a learning model according to claim 1, further comprising: acquiring an indoor image from an imaging device that captures the indoor image including the endoscope operator; and generating the learning model learned so as to output the manipulation information of the next stage in a case where the endoscopic image, the manipulation information, and the indoor image are input to the learning model, based on the training data including the acquired endoscopic image, manipulation information, and indoor image, and the manipulation information of the next stage. ### Claim 6 (depends on 1) The method for generating a learning model according to claim 1, further comprising: acquiring visual line data of the endoscope operator; and generating the learning model learned so as to output the manipulation information and visual line data of the next stage in a case where the endoscopic image, the manipulation information, and the visual line data are input to the learning model, based on the training data including the acquired endoscopic image, manipulation information, and visual line data, and the manipulation information and visual line data of the next stage. ### Claim 7 (depends on 1) The method for generating a learning model according to claim 1, further comprising generating a plurality of types of the learning models according to a manipulation level of the endoscope operator with respect to the manipulation of the endoscope. ### Claim 8 (depends on 1) The method for generating a learning model according to claim 1, further comprising generating a plurality of types of the learning models according to a body type of the endoscope operator. ### Claim 9 (depends on 1) The method for generating a learning model according to claim 1, further comprising: acquiring the manipulation information including a light amount and parameters related to image processing in a processor for an endoscope, the processor acquiring the endoscopic image captured by the endoscope; and generating the learning model learned so as to output the manipulation information including the light amount and the parameters related to the image processing in the next stage, in a case where the endoscopic image and the manipulation information including the light amount and the parameters related to the image processing are input to the learning model. ### Claim 10 A non-transitory computer-readable medium containing a program causing a computer to execute processing comprising: acquiring an endoscopic image captured by an endoscope and manipulation information regarding a manipulation of an endoscope operator in each stage of operation of the endoscope by the endoscope operator operating the endoscope; and inputting the acquired endoscopic image and manipulation information to a learning model learned so as to output the manipulation information of a next stage and outputting the manipulation information of the next stage in a case where the endoscopic image and the manipulation information are input, based on training data including the endoscopic image and manipulation information, and the manipulation information of the next stage; specifying the manipulation level of the endoscope operator with respect to the manipulation of the endoscope; displaying the manipulation information of the next stage, which is output from the learning model, in a case of a first manipulation level according to the specified manipulation level; and in a case of a second manipulation level higher than the first manipulation level, not displaying the manipulation information of the next stage, which is output from the learning model, when predetermined conditions are satisfied, and displaying the manipulation information of the next stage, which is output from the learning model, when the predetermined conditions are not satisfied. ### Claim 11 (depends on 10) The non-transitory computer-readable medium containing a program according to claim 10, causing a computer to execute the processing further comprising: acquiring the manipulation information indicating state data of both arms and both hands of the endoscope operator by using a 3D laser sensor; and inputting, to the learning model, the acquired endoscopic image and the acquired manipulation information indicating the state data of both arms and both hands, and outputting the manipulation information indicating the state data of both arms and both hands of the next stage. ### Claim 12 (depends on 10) The non-transitory computer-readable medium containing a program according to claim 10, causing a computer to execute the processing further comprising: acquiring the manipulation information indicating operation data of a distal end of the endoscope; and inputting, to the learning model, the acquired endoscopic image and the acquired manipulation information indicating the operation data of the distal end of the endoscope, and outputting the manipulation information indicating the operation data of the distal end of the endoscope of the next stage. ### Claim 13 (depends on 10) The non-transitory computer-readable medium containing a program according to claim 10, causing a computer to execute the processing further comprising: causing the learning model to be learned so as to output an image based on the manipulation information indicating the state data of both arms and both hands of the next stage in a case where the endoscopic image and the manipulation information indicating the state data of both arms and both hands are input; acquiring the manipulation information indicating the state data of both arms and both hands of the endoscope operator; and inputting, to the learning model, the acquired endoscopic image and the acquired manipulation information indicating the state data of both arms and both hands, and outputting the image based on the manipulation information indicating the state data of both arms and both hands of the next stage. ### Claim 14 (depends on 10) The non-transitory computer-readable medium containing a program according to claim 10, causing a computer to execute the processing further comprising: acquiring an indoor image from an imaging device capturing the indoor image including the endoscope operator; and inputting the acquired endoscopic image, the acquired manipulation information, and the acquired indoor image to the learning model learned so as to output the manipulation information of the next stage, and outputting the manipulation information of the next stage in a case where the endoscopic image, the manipulation information, and the indoor image are input, based on the training data including the endoscopic image, manipulation information and indoor image, and the manipulation information of the next stage. ### Claim 15 (depends on 10) The non-transitory computer-readable medium containing a program according to claim 10, causing a computer to execute the processing further comprising: acquiring visual line data of the endoscope operator; and inputting the acquired endoscopic image, the acquired manipulation information, and the acquired visual line data to the learning model learned so as to output the manipulation information and visual line data of the next stage, and outputting the manipulation information and visual line data of the next stage in a case where the endoscopic image, the manipulation information, and the visual line data are input, based on the training data including the endoscopic image, manipulation information, and visual line data, and the manipulation information and visual line data of the next stage. ### Claim 16 (depends on 10) The non-transitory computer-readable medium containing a program according to claim 10, causing a computer to execute the processing further comprising: specifying a manipulation level of the endoscope operator with respect to the manipulation of the endoscope; and selecting, from a plurality of types of the learning models prepared according to the manipulation level, the learning model corresponding to the specified manipulation level of the endoscope operator with respect to the manipulation of the endoscope. ### Claim 17 (depends on 10) The non-transitory computer-readable medium containing a program according to claim 10, causing a computer to execute the processing further comprising specifying the manipulation level of the endoscope operator with respect to the manipulation of the endoscope in parallel with output of the manipulation information. ### Claim 18 (depends on 10) The non-transitory computer-readable medium containing a program according to claim 10, causing a computer to execute the processing further comprising: acquiring the indoor image from the imaging device capturing the indoor image including the endoscope operator; specifying, based on the acquired indoor image, the corresponding manipulation information among the manipulation information of the next stage, which is output from the learning model; and displaying the manipulation information of the next stage, which is output from the learning model excluding the specified corresponding manipulation information. ### Claim 19 (depends on 10) The non-transitory computer-readable medium containing a program according to claim 10, causing a computer to execute the processing further comprising outputting screen information for superimposing and displaying the manipulation information of the next stage on a three-dimensional image showing a hollow organ of a subject.