# US 2019/0378293 A1 · Computer Vision Systems and Methods for Real-Time Needle Detection, Enhancement and Localization in Ultrasound

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378293 A1
- Document ID: US20190378293A1
- Status: Published application
- Kind: A1
- Application number: 16/436,284
- Filed: 2019-06-10
- Published: 2019-12-12
- Earliest claimed: 2018-06-08
- Assignees: Rutgers, The State University of New Jersey
- Applicants: Rutgers, The State University of New Jersey
- Inventors: Cosmas Mwikirize; John L. Nosher; Ilker Hacihaliloglu
- Web page: https://patentgrep.com/patent/US20190378293A1

## CPC

- A61B 8/0841: {for locating instruments}
- A61B 5/725: {using specific filters therefor, e.g. Kalman or adaptive filters (specific diagnostics methods using using bioelectric or biomagnetic signals A61B5/316)}
- A61B 5/7267: {involving training the classification device}
- A61B 8/5253: {combining overlapping images, e.g. spatial compounding}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 5/20: using local operators
- G06T 5/70: Denoising; Smoothing
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06T 2207/10132: Ultrasound image
- G06T 2207/20061: Hough transform
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30241: Trajectory
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for detecting placement of an object in a digital image are provided. The system receives a digital image and processes the digital image to generate one or more candidate regions within the digital image using a first neural network. The system then selects a proposed region from the one or more candidate regions using the first neural network and assigns a score to the proposed region using the first neural network. Lastly, the system processes the proposed region using a second neural network to detect an object in the proposed region.

## Claims

### Claim 1

A method for detecting placement of an object in a digital image, comprising:
receiving a digital image;
processing the digital image to generate one or more candidate regions within the digital image using a first neural network;
selecting a proposed region from the one or more candidate regions using the first neural network;
assigning a score to the proposed region using the first neural network; and
processing the proposed region using a second neural network to detect an object in the proposed region.

### Claim 2 (depends on 1)

The method of claim 1, wherein the digital image is an ultrasound image.

### Claim 3 (depends on 2)

The method of claim 2, wherein the score is related to an overlap with ground-truth object information in the ultrasound image.

### Claim 4 (depends on 1)

The method of claim 1, wherein the object is a needle.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first neural network is a regional proposal network (“RPN”).

### Claim 6 (depends on 5)

The method of claim 5, wherein RPN is modeled as a fully convolutional network (“FCN”).

### Claim 7 (depends on 1)

The method of claim 1, wherein the second neural network is a fast region-based convolutional neural network (“R-CNN”).

### Claim 8 (depends on 7)

The method of claim 7, wherein the R-CNN comprises of three convolution layers and two fully connected layers.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating one or more candidate regions comprises ranking potential bounding boxes for the object.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
reducing a number of potential bounding boxes by cross-boundary elimination.

### Claim 11

A method for detecting placement of an object in a digital image, comprising:
detecting a needle in at least one frame of a first set of digital images;
compiling a second set of digital images from the first set, the second set including images in which the needle was detected;
performing a needle tip enhancement process on each frame of the second set of digital images; and
performing a needle tip localization process on each frame of the second set of digital images to detect placement of the needle.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
performing a shaft enhancement process on each frame of the second set of digital images.

### Claim 13 (depends on 11)

The method of claim 11, wherein detecting the needle comprises extracting intensity-invariant local phase descriptors and using the intensity-invariant local phase descriptors to derive histograms of oriented gradients (“HOG”).

### Claim 14 (depends on 11)

The method of claim 11, wherein the needle tip enhancement process comprises removing speckle, reverse attenuation effects, and minimizing the effect of artifacts.

### Claim 15 (depends on 11)

The method of claim 11, wherein the needle tip enhancement process comprises:
performing a patch-wise transmission map process; and
performing a signal transmission map process.

### Claim 16 (depends on 11)

The method of claim 11, wherein the needle tip localization process comprises:
estimating a needle trajectory; and
expanding the needle trajectory over a digital image.

### Claim 17 (depends on 11)

The method of claim 11, wherein the first set of digital images are ultrasound images.

### Claim 18 (depends on 11)

The method of claim 11, wherein the needle tip enhancement process comprises applying high order differential filters.

### Claim 19 (depends on 18)

The method of claim 18, wherein the high order differential filters comprises eight Kirsch filters and one Laplacian filter.

### Claim 20 (depends on 11)

The method of claim 11, wherein the needle tip enhancement process comprises applying a Top-hat filter operation using a linear structure element.

### Claim 21

A system for detecting placement of an object in a digital image, comprising:
a computer system comprising a first neural network and a second neural network, wherein the computer system:
receives a digital image;
processes the digital image to generate one or more candidate regions within the digital image using the first neural network;
selects a proposed region from the one or more candidate regions using the first neural network;
assigns a score to the proposed region using the first neural network; and
processes the proposed region using the second neural network to detect an object in the proposed region.

### Claim 22 (depends on 21)

The system of claim 21, wherein the digital image is an ultrasound image.

### Claim 23 (depends on 22)

The system of claim 22, wherein the score is related to an overlap with ground-truth object information in the ultrasound image.

### Claim 24 (depends on 21)

The system of claim 21, wherein the object is a needle.

### Claim 25 (depends on 21)

The system of claim 21, wherein the first neural network is a regional proposal network (“RPN”).

### Claim 26 (depends on 25)

The system of claim 25, wherein RPN is modeled as a fully convolutional network (“FCN”).

### Claim 27 (depends on 21)

The system of claim 21, wherein the second neural network is a fast region-based convolutional neural network (“R-CNN”).

### Claim 28 (depends on 27)

The system of claim 27, wherein the R-CNN comprises of three convolution layers and two fully connected layers.

### Claim 29 (depends on 21)

The system of claim 21, wherein the first neural network generates the one or more candidate regions by ranking potential bounding boxes for the object.

### Claim 30 (depends on 29)

The system of claim 29, wherein the system reduces a number of potential bounding boxes by cross-boundary elimination.

### Claim 31

A system for detecting placement of an object in a digital image, comprising:
a processor in communication with an digital image source; and
computer system code executed by the processor, the computer system code causing the processor to:
detect a needle in at least one frame of a first set of digital images;
compile a second set of digital images from the first set, the second set including images in which the needle was detected;
perform a needle tip enhancement process on each frame of the second set of digital images; and
perform a needle tip localization process on each frame of the second set of digital images to detect placement of the needle.

### Claim 32 (depends on 31)

The system of claim 31, wherein the computer system code causes the processor to perform a shaft enhancement process on each frame of the second set of digital images.

### Claim 33 (depends on 31)

The system of claim 31, wherein the system detects the needle by extracting intensity-invariant local phase descriptors and using the intensity-invariant local phase descriptors to derive histograms of oriented gradients (“HOG”).

### Claim 34 (depends on 31)

The system of claim 31, wherein the system preforming the needle tip enhancement process comprises removing speckle, reverse attenuation effects, and minimizing the effect of artifacts.

### Claim 35 (depends on 31)

The system of claim 31, wherein the system performing the needle tip enhancement process comprises performing a patch-wise transmission map process and performing a signal transmission map process.

### Claim 36 (depends on 31)

The system of claim 31, wherein the system performing the needle tip localization process comprises estimating a needle trajectory and expanding the needle trajectory over a digital image.

### Claim 37 (depends on 31)

The system of claim 31, wherein the first set of digital images are ultrasound images.

### Claim 38 (depends on 31)

The system of claim 31, wherein the system performing the needle tip enhancement process comprises applying high order differential filters.

### Claim 39 (depends on 38)

The system of claim 38, wherein the high order differential filters comprises eight Kirsch filters and one Laplacian filter.

### Claim 40 (depends on 31)

The system of claim 31, wherein the system performing the needle tip enhancement process comprises applying a Top-hat filter operation using a linear structure element.