# US 12,112,536 B1 · Camera apparatus and method of training and operating neural network model for enhanced foliage detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,536 B1
- Document ID: US12112536B1
- Status: Granted
- Kind: B1
- Application number: 18/401,066
- Filed: 2023-12-29
- Granted: 2024-10-08
- Earliest claimed: 2023-10-19
- Assignees: Tartan Aerial Sense Tech Private Limited
- Applicants: Tartan Aerial Sense Tech Private Limited
- Inventors: Dhivakar Kanagaraj; Pranav M P; Raghul Raghu; Prakash Mathews Pothen
- Web page: https://patentgrep.com/patent/US12112536B1

## CPC

- G06V 20/188: {Vegetation}
- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06T 5/40: using histogram techniques
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G06T 7/11: Region-based segmentation
- G06T 7/136: involving thresholding
- G06T 7/155: involving morphological operators
- G06T 7/194: involving foreground-background segmentation
- G06T 7/90: Determination of colour characteristics
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/273: {removing elements interfering with the pattern to be recognised}
- G06V 10/30: Noise filtering
- G06V 10/56: relating to colour
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- H04N 1/6002: {Corrections within particular colour systems}
- H04N 9/73: Colour balance circuits, e.g. white balance circuits or colour temperature control (camera processing pipelines for colour balance H04N23/88)
- H05K 1/181: associated with surface mounted components
- G06T 2207/10024: Color image
- G06T 2207/20032: Median filtering
- G06T 2207/20036: Morphological image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30188: Vegetation; Agriculture
- G06V 2201/07: Target detection
- H04N 23/84: for processing colour signals
- H05K 2201/10121: Optical component, e.g. opto-electronic component
- H05K 2201/10151: Sensor
- H05K 2201/10159: Memory
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H05K: PRINTED CIRCUITS; CASINGS OR CONSTRUCTIONAL DETAILS OF ELECTRIC APPARATUS; MANUFACTURE OF ASSEMBLAGES OF ELECTRICAL COMPONENTS

## Abstract

A camera apparatus includes control circuitry, where in a training phase, the control circuitry is configured to generate a set of binary images of foliage masks, where each binary image comprises one or more foliage regions demarcated from a non-foliage region. A modified training dataset of color images is generated and a custom neural network model for foliage detection is trained. The control circuitry causes the custom neural network model to not only learn a plurality of features related to foliage but also learn a color variation range of predefined color associated with the plurality of features and utilize combination of the plurality of features related to foliage and the color variation range to obtain trained custom neural network model. The control circuitry is configured to operate the trained custom neural network model to detect one or more foliage regions in a new color image in real time.

## Claims

### Claim 1

A camera apparatus, comprising:
control circuitry, wherein in a training phase, the control circuitry is configured to:
generate a set of binary images of foliage masks, wherein each binary image comprises one or more foliage regions demarcated from a background non-foliage region;
generate a modified training dataset of color images from a first set of input color images; and
train a custom neural network model for foliage detection based on the generated set of binary images of foliage masks and the modified training dataset,
wherein in the training phase, the control circuitry causes the custom neural network model to:
learn a plurality of features related to foliage from the modified training dataset;
further learn a color variation range of a predefined color associated with the plurality of features; and
utilize a combination of the plurality of features related to foliage and the color variation range of the predefined color to obtain a trained custom neural network model, and
wherein in an operational phase, the control circuitry is further configured to:
capture a new color image of an agricultural field;
remove a portion of the new color image, wherein the portion comprises pixels indicative of an artificial object in a field-of-view (FOV) of the camera apparatus; and
operate the trained custom neural network model to detect one or more foliage regions in the new color image in a real time or near real time.

### Claim 2 (depends on 1)

The camera apparatus according to claim 1, wherein the control circuitry, in the training phase, is further configured to:
acquire a first training dataset comprising the first set of input color images of the agricultural field or another agricultural field; and
generate a second set of color images greater in number than the first set of input color images by applying a plurality of image augmentation operations on the first set of input color images, wherein the modified training dataset comprises the first set of input color images and the generated second set of color images.

### Claim 3 (depends on 1)

The camera apparatus according to claim 1, wherein the control circuitry, in the training phase, is further configured to:
configure a convolution neural network model to obtain the custom neural network model by branching the convolution neural network model into a plurality of different types of training branches,
wherein a first type of training branch of the plurality of different types of training branches is configured to learn the plurality of features related to foliage from the modified training dataset,
and wherein a second type of training branch of the plurality of different types of training branches is configured to learn the color variation range of the predefined color associated with the plurality of features.

### Claim 4 (depends on 3)

The camera apparatus according to claim 3, wherein the control circuitry, in the training phase, is further configured to:
integrate outputs from the first type of training branch and the second type of training branch and pass the integrated output to a set of final layers of the convolution neural network model; and
generate an output binary mask image for a corresponding input color image of the modified training dataset,
wherein the output binary mask image is indicative of the one or more foliage regions set in a first binary value different from a second binary value set for the background non-foliage region.

### Claim 5 (depends on 3)

The camera apparatus according to claim 3, wherein the control circuitry, in the training phase, is further configured to assign a different kernel value for each of the plurality of different types of training branches for the configuration of the convolution neural network model.

### Claim 6 (depends on 1)

The camera apparatus according to claim 1, wherein the control circuitry, in the training phase, is further configured to utilize the generated set of binary images of foliage masks as ground truth during training of the custom neural network model.

### Claim 7 (depends on 1)

The camera apparatus according to claim 1, wherein the predefined color is a green color, and wherein the color variation range of the predefined color corresponds to a plurality of different shades of the green color.

### Claim 8 (depends on 1)

The camera apparatus according to claim 1, wherein the new color image of the agricultural field is captured in the FOV ranging from 1.7 to 2.5 meters of the agricultural field.

### Claim 9 (depends on 1)

The camera apparatus according to claim 1, wherein in order to generate the set of binary images of the foliage masks, the control circuitry, in the training phase, is further configured to:
smoothen an input color image of the agricultural filed with a median blur;
convert the smoothened input color image into a plurality of different color spaces; and
execute a set of channel operations on an individual channel or combined channels in each color space of the plurality of different color spaces to enhance green pixels and suppress other pixels, wherein the green pixels are indicative of foliage.

### Claim 10 (depends on 9)

The camera apparatus according to claim 9, wherein in order to generate the set of binary images of the foliage masks, the control circuitry, in the training phase, is further configured to generate a normalized image with the enhanced green pixels based on outputs received from each color space processing path associated with the plurality of different color spaces.

### Claim 11 (depends on 10)

The camera apparatus according to claim 10, wherein in order to generate the set of binary images of the foliage masks, the control circuitry, in the training phase, is further configured to:
determine a threshold value based on a histogram of the normalized image; and
apply the determined threshold value to generate a first binary mask image.

### Claim 12 (depends on 11)

The camera apparatus according to claim 11, wherein in order to generate the set of binary images of the foliage masks, the control circuitry, in the training phase, is further configured to:
apply one or more morphology operations to remove noise in the first binary mask image;
identify image regions that meet a defined criteria to be considered as foliage; and
generate an output binary mask image based on the applied one or more morphology operations and the identified image regions.

### Claim 13 (depends on 12)

The camera apparatus according to claim 12, wherein in order to generate the set of binary images of the foliage masks, the control circuitry, in the training phase, is further configured to generate an optimized binary image of foliage mask by applying filter on the output binary mask image to remove isolated regions and noise, wherein the set of binary images of the foliage masks comprises a number of optimized binary images of the foliage masks.

### Claim 14

A method of training and operating a neural network model for foliage detection, comprising:
in a training phase:
generating a set of binary images of foliage masks, wherein each binary image comprises one or more foliage regions demarcated from a background non-foliage region;
generating a modified training dataset of color images; and
training a custom neural network model for foliage detection based on the generated set of binary images of foliage masks and the modified training dataset; and
causing the custom neural network model in the training phase to:
learn a plurality of features related to foliage from the modified training dataset;
further learn a color variation range of a predefined color associated with the plurality of features; and
utilize a combination of the plurality of features related to foliage and the color variation range of the predefined color to obtain a trained custom neural network model, and
in an operational phase:
capturing a new color image of an agricultural field;
removing a portion of the new color image, wherein the portion comprises pixels indicative of an artificial object in a field-of-view (FOV) of the camera apparatus; and
operating the trained custom neural network model to detect one or more foliage regions in the new color image in a real time or near real time after removal of the portion.

### Claim 15 (depends on 14)

The method according to claim 14, further comprising in the training phase:
acquiring a first training dataset comprising a first set of input color images of the agricultural field or another agricultural field; and
generating a second set of color images greater in number than the first set of input color images by applying a plurality of image augmentation operations on the first set of input color images, wherein the modified training dataset comprises the first set of input color images and the generated second set of color images.

### Claim 16 (depends on 14)

The method according to claim 14, further comprising in the training phase:
configuring a convolution neural network model to obtain the custom neural network model by branching the convolution neural network model into a plurality of different types of training branches,
wherein a first type of training branch of the plurality of different types of training branches is configured to learn the plurality of features related to foliage from the modified training dataset,
and wherein a second type of training branch of the plurality of different types of training branches is configured to learn the color variation range of the predefined color associated with the plurality of features.

### Claim 17 (depends on 16)

The method according to claim 16, further comprising in the training phase:
integrating outputs from the first type of training branch and the second type of training branch and pass the integrated output to a set of final layers of the convolution neural network model; and
generating an output binary mask image for a corresponding input color image of the modified training dataset,
wherein the output binary mask image is indicative of the one or more foliage regions set in a first binary value different from a second binary value set for the background non-foliage region.

### Claim 18 (depends on 16)

The method according to claim 16, further comprising in the training phase:
assigning a different kernel value for each of the plurality of different types of training branches for the configuration of the convolution neural network model.

### Claim 19 (depends on 14)

The method according to claim 14, further comprising utilizing, in the training phase, the generated set of binary images of foliage masks as ground truth during training of the custom neural network model.

### Claim 20 (depends on 14)

The method according to claim 14, wherein the predefined color is a green color, and wherein the color variation range of the predefined color corresponds to a plurality of different shades of the green color.