# US 12,008,758 B2 · Detection of polyps > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,758 B2 - Document ID: US12008758B2 - Status: Granted - Kind: B2 - Application number: 17/952,234 - Filed: 2022-09-24 - Published: 2023-10-19 - Granted: 2024-06-11 - Earliest claimed: 2016-05-19 - Assignees: Psip LLC - Applicants: Psip LLC - Inventors: Salmaan Hameed; Giau Nguyen - Web page: https://patentgrep.com/patent/US12008758B2 ## CPC - G06T 7/0014: {using an image reference approach} - A61B 1/000094: {extracting biological structures} - A61B 1/000096: {using artificial intelligence} - A61B 1/0014: {Fastening element for attaching accessories to the outside of an endoscope, e.g. clips, clamps or bands} - A61B 1/00177: {for 90 degrees side-viewing} - A61B 1/00181: {for multiple fixed viewing angles (A61B1/00193 takes precedence)} - G01B 11/03: by measuring coordinates of points - G01B 11/30: for measuring roughness or irregularity of surfaces - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - A61B 1/00101: {the distal tip features being detachable} - G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]} - G06T 2207/10068: Endoscopic image - G06T 2207/30032: Colon polyp - G06V 2201/032: of protuberances, polyps nodules, etc. - G06V 2201/034: of medical instruments - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Identifying polyps or lesions in a colon. In some variations, computer-implemented methods for polyp detection may be used in conjunction with an endoscope system to analyze the images captured by the endoscopic system, identify any polyps and/or lesions in a visual scene captured by the endoscopic system, and provide an indication to the practitioner that a polyp and/or lesion has been detected. ## Claims ### Claim 1 A method for lesion detection, comprising the steps of: in an image of a body region acquired from an imaging module located at a distal portion of an endoscope, by processor of a computer, identifying surface peaks in the acquired image, the surface peaks identified via histograms of oriented gradients (HOG) and/or maximally stable extremal region (MSER); by processor of the computer, identifying regions of interest around the one or more identified surface peaks based on one or more of clustering of surface peaks, a separation distance, surface peak density, high intensity pixel densities, size and shape of high-contrast edges, spatial frequency of low-contrast edges, RGB values, and/or changes of RGB values across a region; by processor of the computer, within the region(s) of interest, identifying features likely to be lesions of interest by applying a convolutional neural network (CNN) designed to compare the region(s) of interest to with a plurality of images containing lesions and a plurality of images that do not contain lesions; and for features likely to be lesions of interest based on application of the CNN, computing an indication of the identification of the identified likelihood. ### Claim 2 (depends on 1) The method of claim 1, wherein: identifying the surface peaks comprises computing a histogram of oriented gradients. ### Claim 3 (depends on 1) The method of claim 1, wherein: the identifying features likely to be lesions of interest comprises identifying a curvature of a high-contrast edge. ### Claim 4 (depends on 1) The method of claim 1, wherein: wherein the endoscope is a colonoscope, the body region is a colon of a patient, and the lesions of interest are colon polyps. ### Claim 5 (depends on 4) The method of claim 4: wherein applying a convolutional neural network (CNN) to the region(s) of interest further comprises: generating a first filtered pixel region by filtering the pixel region with a first filter to identify one or more polyp-like features, generating a second filtered pixel region by filtering the first filtered pixel region with a second filter to identify one or more non-polyp features, and further comprising generating a notification that a polyp has been detected based on whether a one of the region(s) of interest has been identified that has a higher incidence of polyp-like features than non-polyp features. ### Claim 6 (depends on 1) The method of claim 1: wherein the imaging module comprises a first side-facing imaging device and a second side-facing imaging device, and wherein the image has been acquired from a first image from the first side-facing imaging device and a second image from the second side-facing imaging device. ### Claim 7 (depends on 1) The method of claim 1, further comprising: by processor of a computer, identifying regions from around the region(s) of interest where a ratio of the incidence of the polyp-like feature to the incidence of the non-polyp feature exceeds a pre-determined threshold. ### Claim 8 (depends on 1) The method of claim 1, wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on surface peak density. ### Claim 9 (depends on 7) The method of claim 7, wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on a radius of curvature of a high contrast edge. ### Claim 10 (depends on 9) The method of claim 9 wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on a location of a pixel having a local maximum intensity relative to an inner curve of the high-contrast edge. ### Claim 11 (depends on 1) The method of claim 1, wherein: the polyp-like feature comprises surface peaks identified by calculating a histogram of oriented gradients of a plurality of polyp-positive colon images (HOG-PI). ### Claim 12 (depends on 1) The method of claim 1, wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on spatial frequency of low-contrast edges. ### Claim 13 (depends on 1) The method of claim 1, wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on change of RGB values across the region(s) of interest around the one or more identified surface peaks. ### Claim 14 (depends on 1) The method of claim 1: wherein the endoscope has multiple imaging devices, and the method further comprises identifying surface peaks in images from the multiple imaging devices via histograms of oriented gradients and applying the CNN to regions of interest identified in images from the multiple imaging devices. ### Claim 15 (depends on 1) The method of claim 1: wherein applying a convolutional neural network (CNN) to the region(s) of interest further comprises: generating a first filtered pixel region by filtering the pixel region with a first filter to identify one or more lesion-like features, generating a second filtered pixel region by filtering the first filtered pixel region with a second filter to identify one or more non-lesion features, and further comprising generating a notification that a lesion has been detected based on whether a one of the region(s) of interest has been identified that has a higher incidence of lesion-like features than non-lesion features. ### Claim 16 An apparatus, comprising: a processor, and a non-transitory memory having stored therein one or more programs programmed to cause the processor to: in an image of a body region acquired from an imaging module located at a distal portion of an endoscope, identify surface peaks in the image, the surface peaks identified via histograms of oriented gradients (HOG) and/or maximally stable extremal region (MSER); identify regions of interest around the one or more identified surface peaks based on one or more of clustering of surface peaks, a separation distance, surface peak density, high intensity pixel densities, size and shape of high-contrast edges, spatial frequency of low-contrast edges, RGB values, and/or changes of RGB values across a region; within the region(s) of interest, identify features likely to be lesions of interest by applying a convolutional neural network (CNN) designed to compare the region(s) of interest to with a plurality of images containing lesions and a plurality of images that do not contain lesions; and for features likely to be lesions of interest based on application of the CNN, compute an indication of the identification of the identified likelihood. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the programs are further programmed to: identify the surface peaks by computing a histogram of oriented gradients. ### Claim 18 (depends on 16) The apparatus of claim 16, wherein the programs are further programmed to: apply a convolutional neural network (CNN) to the region(s) of interest based on training of the CNN on images with lesions and images without lesions. ### Claim 19 (depends on 16) The apparatus of claim 16, wherein: wherein the endoscope is a colonoscope, the body region is a colon of a patient, and the lesions of interest are colon polyps. ### Claim 20 (depends on 16) The apparatus of claim 16, wherein the programs are further programmed to cause the processor to: identify features likely to be lesions of interest based at least in part on identifying a curvature of a high-contrast edge. ### Claim 21 (depends on 16) The apparatus of claim 16, wherein: wherein the endoscope is a colonoscope, the body region is a colon of a patient, and the lesions of interest are colon polyps. ### Claim 22 (depends on 16) The apparatus of claim 16: wherein the programs to apply a convolutional neural network (CNN) to the region(s) of interest are further programmed to cause the processor to: generate a first filtered pixel region by filtering the pixel region with a first filter to identify one or more polyp-like features, generate a second filtered pixel region by filtering the first filtered pixel region with a second filter to identify one or more non-polyp features, and the programs are further programmed to cause the processor to generate a notification that a polyp has been detected based on whether a one of the region(s) of interest has been identified that has a higher incidence of polyp-like features than non-polyp features. ### Claim 23 (depends on 16) The apparatus of claim 16: wherein the imaging module comprises a first side-facing imaging device and a second side-facing imaging device, and wherein the image has been acquired from a first image from the first side-facing imaging device and a second image from the second side-facing imaging device. ### Claim 24 (depends on 16) The apparatus of claim 16, the programs being further programmed to cause the processor to: by processor of a computer, to identify regions from around the region(s) of interest where a ratio of the incidence of the polyp-like feature to the incidence of the non-polyp feature exceeds a pre-determined threshold. ### Claim 25 (depends on 16) The apparatus of claim 16, wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on surface peak density. ### Claim 26 (depends on 16) The apparatus of claim 16, wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on a radius of curvature of a high contrast edge. ### Claim 27 (depends on 16) The apparatus of claim 16 wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on a location of a pixel having a local maximum intensity relative to an inner curve of the high-contrast edge. ### Claim 28 (depends on 16) The apparatus of claim 16, wherein: the polyp-like feature comprises surface peaks identified by calculating a histogram of oriented gradients of a plurality of polyp-positive colon images (HOG-PI). ### Claim 29 (depends on 16) The apparatus of claim 16, wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on spatial frequency of low-contrast edges. ### Claim 30 (depends on 16) The apparatus of claim 16, wherein: the identifying of regions of interest around the one or more identified surface peaks is based, as least in part, on change of RGB values across the region(s) of interest around the one or more identified surface peaks. ### Claim 31 (depends on 16) The apparatus of claim 16 : wherein the endoscope has multiple imaging devices, and the programs are further programmed to cause the processor to identify surface peaks in images from the multiple imaging devices via histograms of oriented gradients and apply the CNN to regions of interest identified in images from the multiple imaging devices. ### Claim 32 (depends on 16) The apparatus of claim 16: the programs are further programmed to cause the processor to apply a convolutional neural network (CNN) to the region(s) of interest are further programmed to cause the processor to: generate a first filtered pixel region by filtering the pixel region with a first filter to identify one or more lesion-like features, generate a second filtered pixel region by filtering the first filtered pixel region with a second filter to identify one or more non-lesion features, and programs are further programmed to cause the processor to generate a notification that a lesion has been detected based on whether a one of the region(s) of interest has been identified that has a higher incidence of lesion-like features than non-lesion features.