# US 12,008,816 B2 · Method and system for real time object detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,816 B2
- Document ID: US12008816B2
- Status: Granted
- Kind: B2
- Application number: 17/540,172
- Filed: 2021-12-01
- Published: 2022-06-09
- Granted: 2024-06-11
- Earliest claimed: 2020-12-03
- Assignees: Hl Klemove Corp.
- Applicants: HL Klemove Corp.
- Inventors: Vipin Sharma; Akshat Agrawal; Jitesh Kumar Singh; Arpit Awasthi
- Web page: https://patentgrep.com/patent/US12008816B2

## CPC

- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06F 18/24: Classification techniques
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/32: Normalisation of the pattern dimensions
- G06V 10/40: Extraction of image or video features
- G06V 10/82: using neural networks
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure relates to a method for real-time object detection, the method comprising: capturing an image in vicinity of a vehicle; feeding the captured image to a deep fully convolution neural network; extracting one or more relevant features from the captured image; classifying the extracted features using one or more branches to identify different size of objects; predicting objects present in the image based on a predetermined confidence threshold; and marking the predicted objects in the image.

## Claims

### Claim 1

A method for real-time object detection for a host vehicle, the method comprising:
capturing an image in vicinity of the host vehicle;
feeding the captured image to a deep fully convolution neural network;
extracting one or more relevant features from the captured image;
classifying the extracted features using one or more branches of the deep fully convolution neural network to identify different size of objects, each of the one or more branches comprising a different receptive field corresponding to the size of the object;
predicting objects present in the image based on a predetermined confidence threshold;
marking the predicted objects in the image; and
plotting the marked image on a display,
wherein the different receptive field is created by performing a different number of down sampling and a different number of depth-wise separable convolution to the extracted features.

### Claim 2 (depends on 1)

The method of claim 1, wherein the captured image is a Ground Truth (GT) image marked using a Bounding Box annotation tool.

### Claim 3 (depends on 1)

The method of claim 1, further comprising reshaping the captured images into a predetermined compatible size, while still maintaining the aspect ratio of the objects present in the image which in turn is fed to the deep fully convolution neural network.

### Claim 4 (depends on 1)

The method of claim 1, wherein classifying includes routing the object having a smaller size early off for the prediction in the deep fully convolution neural network.

### Claim 5 (depends on 1)

The method of claim 1, wherein the down sampling comprises:
down sampling a feature map, wherein the feature map is extracted from an image from a plurality of feature vectors, to record the most activated features of the image; and
simultaneously down sampling the same feature map to maintain a localization information,
wherein the deep fully convolution neural network comprises down sampling-convolution-receptive block (DCR) technique.

### Claim 6 (depends on 1)

The method of claim 1, wherein the predicting objects comprises:
comparing a confidence score associated with each of intermediary prediction objects to the predetermined confidence threshold; and
choosing predicted objects having the score above or equal to the predetermined confidence threshold.

### Claim 7

A system for real-time object detection for a host vehicle, comprising:
an image sensor configured to capture an image in the vicinity of the host vehicle;
a controller communicatively connected to the image sensor and configured to:
obtain the captured from the image sensor;
feed the captured image to a deep fully convolution neural network;
extract one or more relevant features from the captured image;
classify the extracted features using one or more branches of the deep fully convolution neural network to identify different size of objects, each of the or more branches comprising a different receptive field corresponding to the size of the object;
predict objects present in the image based on a predetermined confidence threshold;
mark the predicted objects in the image; and
plot the marked image on a display,
wherein the different receptive field is created by performing a different number of down sampling and a different number of depth-wise separable convolution to the extracted features.

### Claim 8 (depends on 7)

The system of claim 7, wherein the controller is further configured to reshape the captured images into a predetermined compatible size while still maintaining the aspect ratio of the objects present in the image to be fed to the deep fully convolution neural network.

### Claim 9 (depends on 7)

The system of claim 7, wherein the controller is further configured to route the object having a smaller size early off, for the prediction in the deep fully convolution neural network.

### Claim 10 (depends on 7)

The system of claim 7, wherein the down sampling comprises:
down sampling a feature map, wherein the feature map is extracted from an image from a plurality of feature vectors, to record the most activated features of the image; and
simultaneously down sampling the same feature map to maintain a localization information,
wherein the deep fully convolution neural network comprises down sampling-convolution-receptive block (DCR) technique.

### Claim 11 (depends on 7)

The system for object detection of claim 7, wherein the controller is further configured to:
compare a confidence score associated with each of intermediary prediction objects to the predetermined confidence threshold; and
choose predicted objects having the score above or equal to the predetermined confidence threshold.