# US 12,602,779 B2 · Image processing device, image processing method and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,779 B2 - Document ID: US12602779B2 - Status: Granted - Kind: B2 - Application number: 18/278,294 - Filed: 2021-02-26 - Published: 2024-04-18 - Granted: 2026-04-14 - Earliest claimed: 2021-02-26 - Assignees: NEC Corporation - Applicants: NEC Corporation - Inventors: Masahiro Saikou - Web page: https://patentgrep.com/patent/US12602779B2 ## CPC - G06T 7/0012: {Biomedical image inspection} - A61B 1/000094: {extracting biological structures} - A61B 1/000096: {using artificial intelligence} - G06T 7/11: Region-based segmentation - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis - G06V 10/52: Scale-space analysis, e.g. wavelet analysis (multi-scale boundary representations G06V10/42) - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/762: using clustering, e.g. of similar faces in social networks - G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space - G06V 10/806: {of extracted features} - G06V 10/82: using neural networks - G06T 2207/10068: Endoscopic image - G06V 2201/03: Recognition of patterns in medical or anatomical images - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The image processing device 1X includes an acquisition means 31X, a mask image generation means 32X, and an identification means 33X. The acquisition means 31X acquires a captured image acquired by photographing an inspection target by a photographing unit provided in an endoscope. The mask image generation means 32X generates a plurality of mask images which indicate candidate areas of an attention part in the captured image by different levels of granularity. The identification means 33X identifies the attention part based on the plurality of the mask images. The image processing device is able to support decision-making based on identification results of the attention part in endoscopic examination. ## Claims ### Claim 1 An image processing device, comprising: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to: acquire a captured image acquired by photographing an inspection target by a photographing unit provided in an endoscope; generate a plurality of mask images which indicate candidate areas of an attention part in the captured image by different levels of granularity; and identify the attention part based on the plurality of the mask images, wherein the plurality of mask images are images each indicating a reliability of being the attention part for each of grids corresponding to a plurality of pixels in the captured image, and wherein numbers of the grids determined according to the levels of granularity are different among the plurality of the mask images. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to generate a plurality of feature maps having different scales based on the captured image and generates the plurality of the mask images from the plurality of the feature maps, respectively. ### Claim 3 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to identify the attention point based on a mask image selected from the plurality of the mask images. ### Claim 4 (depends on 3) The image processing device according to claim 3, wherein the at least one processor is configured to execute the instructions to calculate degrees of similarity between the plurality of the mask images and determines a mask image to be selected or integrated based on the degrees of similarity. ### Claim 5 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to identify the attention point based on a mask image into which the plurality of the mask images are integrated. ### Claim 6 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is configured to further execute the instructions to output information relating to the attention part. ### Claim 7 (depends on 6) The image processing device according to claim 6, wherein the at least one processor is configured to execute the instructions to output, as the information relating to the attention part, at least one of information relating to presence or absence of the attention part or information relating to a position of the attention part in the captured image. ### Claim 8 (depends on 6) The image processing device according to claim 6, wherein the at least one processor is configured to execute the instructions to convert an image in which the attention part is identified into a granularity adjustment image that is an image indicating the attention part by a specified level of granularity, wherein the at least one processor is configured to execute the instructions to display, on a display device, the granularity adjustment image or an image based on the granularity adjustment image. ### Claim 9 (depends on 1) The image processing device according to claim 1, wherein the attention part is a biopsy site. ### Claim 10 (depends on 1) The image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to generate the plurality of the mask images by using a mask image generation model trained through machine learning with training data including captured images and a plurality of mask images. ### Claim 11 An image processing method executed by a computer, the image processing method comprising: acquiring a captured image acquired by photographing an inspection target by a photographing unit provided in an endoscope; generating a plurality of mask images which indicate candidate areas of an attention part in the captured image by different levels of granularity; and identifying the attention part based on the plurality of the mask images, wherein the plurality of mask images are images each indicating a reliability of being the attention part for each of grids corresponding to a plurality of pixels in the captured image, and wherein numbers of the grids determined according to the levels of granularity are different among the plurality of the mask images. ### Claim 12 (depends on 11) The image method device according to claim 11, further comprising generating a plurality of feature maps having different scales based on the captured image and generates the plurality of the mask images from the plurality of the feature maps, respectively. ### Claim 13 (depends on 11) The image processing method according to claim 11, further comprising identifying the attention point based on a mask image selected from the plurality of the mask images. ### Claim 14 (depends on 11) The image processing method according to claim 11, further comprising identifying the attention point based on a mask image into which the plurality of the mask images are integrated. ### Claim 15 (depends on 14) The image processing method according to claim 14, further comprising calculating degrees of similarity between the plurality of the mask images and determines a mask image to be selected or integrated based on the degrees of similarity. ### Claim 16 (depends on 11) The image processing method according to claim 11, further comprising outputting information relating to the attention part. ### Claim 17 (depends on 16) The image processing method according to claim 16, further comprising outputting, as the information relating to the attention part, at least one of information relating to presence or absence of the attention part or information relating to a position of the attention part in the captured image. ### Claim 18 (depends on 16) The image processing device according to claim 16, further comprising converting an image in which the attention part is identified into a granularity adjustment image that is an image indicating the attention part by a specified level of granularity, and displaying, on a display device, the granularity adjustment image or an image based on the granularity adjustment image. ### Claim 19 (depends on 11) The image processing method according to claim 11, wherein the attention part is a biopsy site. ### Claim 20 A non-transitory computer readable storage medium storing a program executed by a computer, the program causing the computer to: acquire a captured image acquired by photographing an inspection target by a photographing unit provided in an endoscope; generate a plurality of mask images which indicate candidate areas of an attention part in the captured image by different levels of granularity; and identify the attention part based on the plurality of the mask images, wherein the plurality of mask images are images each indicating a reliability of being the attention part for each of grids corresponding to a plurality of pixels in the captured image, and wherein numbers of the grids determined according to the levels of granularity are different among the plurality of the mask images.