# US 11,443,421 B2 · Method and system for detecting infestation in agricultural products > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,421 B2 - Document ID: US11443421B2 - Status: Granted - Kind: B2 - Application number: 17/186,252 - Filed: 2021-02-26 - Published: 2022-07-14 - Granted: 2022-09-13 - Earliest claimed: 2021-01-12 - Assignees: Wipro Limited - Applicants: Wipro Limited - Inventors: Sarthak Panigrahi; Anandaraj Thangappan - Web page: https://patentgrep.com/patent/US11443421B2 ## CPC - G06T 7/001: {using an image reference approach} - G01B 11/22: for measuring depth - G01J 3/0289: {Field-of-view determination; Aiming or pointing of a spectrometer; Adjusting alignment; Encoding angular position; Size of measurement area; Position tracking} - G01J 3/2823: {Imaging spectrometer} - G01N 21/84: Systems specially adapted for particular applications - G01N 21/94: Investigating contamination, e.g. dust (G01N21/85 takes precedence) - G01N 23/04: and forming images of the material - G01N 23/083: the radiation being X-rays - G01N 33/025: {Fruits or vegetables} - G01S 17/89: for mapping or imaging - G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2413: based on distances to training or reference patterns - G06F 18/25: Fusion techniques - G06T 7/0004: {Industrial image inspection} - G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model - G01J 2003/2826: {Multispectral imaging, e.g. filter imaging} - G01J 2003/2833: {and memorised spectra collection} - G01N 2021/8466: {Investigation of vegetal material, e.g. leaves, plants, fruits} - G01N 2223/04: and measuring absorption - G01N 2223/40: Imaging - G01N 2223/618: food - G01N 2223/646: flaws, defects - G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10036: Multispectral image; Hyperspectral image - G06T 2207/10116: X-ray image - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30128: Food products - G06V 10/58: relating to hyperspectral data - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N) - G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract System and method of detecting infestation in agricultural products is disclosed. In one embodiment, an infestation detection system captures hyperspectral imaging data and depth imaging data for a plurality of points on an agricultural product upon directing a light source at the agricultural product. The system analyses the captured imaging data to derive morphological details as well as spectral signatures for complete 360° view of the plurality of points on the agricultural product. In an embodiment, the spectral signatures may be corrected for one or more pixels associated with the plurality of points in the agricultural product by integrating the hyperspectral imaging data with the depth imaging data. The system further classifies one or more regions of the agricultural product based on matching of spectral signatures for the one or more pixels with pre-defined spectral signatures stored for a plurality of agricultural products, and accordingly detect infestation. ## Claims ### Claim 1 A method of detecting infestation in agricultural products, the method comprises: capturing, by an infestation detection system, hyperspectral imaging data and depth imaging data for a plurality of points on an agricultural product upon directing a light source at the plurality of points on the agricultural product; deriving, by the infestation detection system, morphological details for the plurality of points on the agricultural product; determining, by the infestation detection system, spectral signatures for one or more pixels associated with the plurality of points in the agricultural product for complete 360° view by integrating the hyperspectral imaging data with the depth imaging data, wherein determining the spectral signatures further comprises: correcting spectrum for one or more pixels associated with the plurality of points in the agricultural product based on depth information associated with the one or more pixels and a transformation function computed by integrating the hyperspectral imaging data with the depth imaging data, wherein the transformation function is computed for the plurality of points based on change in intensity over change in angle between a first point at top plane of the agricultural product to a second at any other region than top plane of the agricultural product; and classifying, by the infestation detection system, one or more regions of the agricultural product based on matching of the spectral signatures determined for the one or more pixels with pre-defined spectral signatures stored for a plurality of agricultural products in an infestation knowledge database, and thereby detecting the infestation in the agricultural product. ### Claim 2 (depends on 1) The method as claimed in claim 1 further comprises segregating the agricultural product into at least one of fresh, infested or severely damaged based on the infestation detected in the one or more regions of the agricultural product using the spectral signatures. ### Claim 3 (depends on 1) The method as claimed in claim 1, wherein classifying the one or more regions of the agricultural product further comprises: identifying one or more morphological regions in the agricultural product using a pre-trained model; deriving the spectral signatures by integrating the depth imaging data with corresponding hyperspectral imaging data for the one or more morphological regions; and detecting the infestation for the one or more morphological regions in the agricultural product. ### Claim 4 (depends on 1) The method as claimed in claim 1, wherein the hyperspectral imaging data comprises spectral information associated with one or more pixels obtained using hyperspectral imaging in the form of hyperspectral cubes for one or more pixels in the agricultural product; and wherein the depth imaging data comprises depth information for one or more pixels associated with the one or more regions in the agricultural product obtained using depth sensor. ### Claim 5 (depends on 1) The method as claimed in claim 1, wherein detecting the infestation in the agricultural product further comprises determining at least one of a type of the agricultural product and associated one or more disease type, stages of infestation, severity of damages, presence of pathogens, presence of harmful residues, survival life of the agricultural product, probable treatment against infestation. ### Claim 6 (depends on 1) The method as claimed in claim 1, wherein detecting the infestation in the agricultural product further comprises detecting internal damage in the agricultural product using x-ray imaging data. ### Claim 7 An infestation detection system for agricultural products, the infestation detection system comprises: a plurality of image sensors; at least one processor; a memory communicatively coupled to the at least one processor, wherein the memory stores the at least one processor-executable instructions, which, on execution, causes the at least one processor to: capture hyperspectral imaging data and depth imaging data for a plurality of points on an agricultural product upon directing a light source at the plurality of points on the agricultural product; derive morphological details for the plurality of points on the agricultural product; determine spectral signatures for one or more pixels associated with the plurality of points in the agricultural product for complete 360° view by integrating the hyperspectral imaging data with the depth imaging data, wherein the at least one processor-executable instructions to determine the spectral signatures further causes the at least one processor-executable to: correct spectrum for one or more pixels associated with the plurality of points in the agricultural product based on depth information associated with the one or more pixels and a transformation function computed by integrating the hyperspectral imaging data with the depth imaging data, wherein the transformation function is computed for the plurality of points based on change in intensity over change in angle between a first point at top plane of the agricultural product to a second at any other region than top plane of the agricultural product; and classify one or more regions of the agricultural product based on matching of spectral signatures determined for the one or more pixels with pre-defined spectral signatures stored for a plurality of agricultural products in an infestation knowledge database, and thereby detecting infestation in the agricultural product. ### Claim 8 (depends on 7) The infestation detection system as claimed in claim 7 further causes the at least one processor to segregate the agricultural product into at least one of fresh, infested or severely damaged based on the infestation detected in the one or more regions of the agricultural product using the spectral signatures. ### Claim 9 (depends on 7) The infestation detection system as claimed in claim 7, wherein the at least one processor-executable instructions to classify the one or more regions of the agricultural product further causes the at least one processor to: identify one or more morphological regions in the agricultural product using a pre-trained model; derive the spectral signatures by integrating the depth imaging data with corresponding hyperspectral imaging data for the one or more morphological regions: and detect the infestation for the one or more morphological regions in the agricultural product. ### Claim 10 (depends on 7) The infestation detection system as claimed in claim 7, wherein the hyperspectral imaging data comprises spectral information associated with one or more pixels obtained using hyperspectral imaging in the form of hyperspectral cubes for one or more pixels in the agricultural product; and wherein the depth imaging data comprises depth information for one or more pixels associated with the one or more regions in the agricultural product obtained using depth sensor. ### Claim 11 (depends on 7) The infestation detection system as claimed in claim 7, wherein the at least one processor-executable instructions to detect infestation in the agricultural product further causes the at least one processor to determine at least one of a type of the agricultural product and associated one or more disease types, stages of infestation, severity of damages, presence of pathogens, presence of harmful residues, survival life of the agricultural product, probable treatment against infestation. ### Claim 12 (depends on 7) The infestation detection system as claimed in claim 7, wherein the at least one processor-executable instructions to detect infestation in the agricultural product further causes the at least one processor to detect internal damage in the agricultural product using x-ray imaging data. ### Claim 13 A non-transitory computer-readable medium having stored thereon a set of computer-executable instructions for detecting infestation in agricultural products, the set of computer-executable instructions causing a computer comprising one or more processors to perform steps comprising: capturing, by an infestation detection system, hyperspectral imaging data and depth imaging data for a plurality of points on an agricultural product upon directing a light source at the plurality of points on the agricultural product; deriving, by the infestation detection system, morphological details for the plurality of points on the agricultural product; determining, by the infestation detection system, spectral signatures for one or more pixels associated with the plurality of points in the agricultural product for complete 360° view by integrating the hyperspectral imaging data with the depth imaging data, wherein determining the spectral signatures further comprises: correcting spectrum for one or more pixels associated with the plurality of points in the agricultural product based on depth information associated with the one or more pixels and a transformation function computed by integrating the hyperspectral imaging data with the depth imaging data, wherein the transformation function is computed for the plurality of points based on change in intensity over change in angle between a first point at top plane of the agricultural product to a second at any other region than top plane of the agricultural product; and classifying, by the infestation detection system, one or more regions of the agricultural product based on matching of the spectral signatures determined for the one or more pixels with pre-defined spectral signatures stored for a plurality of agricultural products in an infestation knowledge database, and thereby detecting the infestation in the agricultural product. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the steps further comprise segregating the agricultural product into at least one of fresh, infested or severely damaged based on the infestation detected in the one or more regions of the agricultural product using the spectral signatures. ### Claim 15 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein classifying the one or more regions of the agricultural product further comprises: identifying one or more morphological regions in the agricultural product using a pre-trained model; deriving the spectral signatures by integrating the depth imaging data with corresponding hyperspectral imaging data for the one or more morphological regions; and detecting the infestation for the one or more morphological regions in the agricultural product. ### Claim 16 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the steps for detecting infestation in the agricultural product further comprise determining at least one of a type of the agricultural product and associated one or more disease type, stages of infestation, severity of damages, presence of pathogens, presence of harmful residues, survival life of the agricultural product, probable treatment against infestation. ### Claim 17 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the steps for detecting infestation in the agricultural product further comprise detecting internal damage in the agricultural product using x-ray imaging data.