# US 2017/0287124 A1 · System and method for detecting missing tooth in mining shovel > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0287124 A1 - Document ID: US20170287124A1 - Status: Published application - Kind: A1 - Application number: 15/090,646 - Filed: 2016-04-05 - Published: 2017-10-05 - Earliest claimed: 2016-04-05 - Assignees: General Electric Company - Applicants: General Electric Company - Inventors: Ser Nam Lim; Ning Zhou; Joao Vitor Baldini Soares - Web page: https://patentgrep.com/patent/US20170287124A1 ## CPC - G06T 7/001: {using an image reference approach} - E02F 9/267: {Diagnosing or detecting failure of vehicles} - E02F 9/2808: {Teeth} - G06T 7/248: {involving reference images or patches} - G06T 7/254: involving subtraction of images - G06T 7/269: using gradient-based methods - G06T 7/74: {involving reference images or patches} - G06T 7/90: Determination of colour characteristics - G06T 2207/10016: Video; Image sequence - G06T 2207/30164: Workpiece; Machine component - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - E02F: DREDGING; SOIL-SHIFTING (winning peat E21C49/00) ## Abstract A method for detecting missing tooth in mining shovel, implemented using a processing device, includes receiving a pair of image frames from a camera disposed on a rope mine shovel configured to carry a mining load. A tooth line region corresponding to the pair of image frames is detected to generate a pair of tooth line regions based on a shovel template set. A difference image is determined based on the pair of image frames and the pair of tooth line regions. Further, a response map representative of possible tooth positions is determined based on the difference image using a tooth template matching technique. A tooth line is selected among a plurality of candidate tooth lines based on the response map. Further, a tooth condition is determined based on the tooth line and the difference image. The tooth condition is notified to an operator of the rope mine shovel. ## Claims ### Claim 1 A method, comprising: receiving a pair of image frames from a camera disposed on a rope mine shovel configured to carry a mining load, wherein the pair of image frames comprises a first image frame and a second image frame; detecting a corresponding tooth line region for the pair of image frames to generate a pair of tooth line regions based on a shovel template set, wherein the shovel template set comprises a plurality of bucket template images, a plurality of tooth line images, a plurality of locations corresponding to the plurality of bucket template images, and a plurality of locations of tooth line images, and wherein the pair of tooth line regions comprises a first tooth line region in the first image and a second tooth line region in the second image; determining a difference image based on the pair of image frames and the pair of tooth line regions; determining a response map representative of possible tooth positions based on the difference image using a tooth template matching technique; selecting a tooth line among a plurality of candidate tooth lines based on the response map; determining a tooth condition based on the tooth line and the difference image and notifying the tooth condition to an operator of the rope mine shovel. ### Claim 2 (depends on 1) The method of claim 1, wherein notifying the tooth condition comprises emitting an alarm. ### Claim 3 (depends on 1) The method of claim 1, wherein detecting the tooth line region comprises: generating a plurality of image tiles corresponding to an image frame among the pair of image frames; determining a correlation between a bucket template image among a plurality of bucket template images and an image tile among the plurality of image tiles to generate a plurality of matching scores; selecting a bucket template image and an associated image tile corresponding to a minimum value among the plurality of matching scores; and determining a location of the tooth line region based on location of the selected image tile, location of the selected bucket template image, and a location of tooth line region associated with the selected bucket template image. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the difference image comprises: determining a homography transformation of the first tooth line image with respect to the second tooth line image; and generating a transformed image by applying the homography transformation to the first image. ### Claim 5 (depends on 3) The method of claim 3, wherein determining the difference image comprises: determining a first gradient image corresponding to the transformed image and a second gradient image corresponding to the second image frame; and performing a differencing operation between the first gradient image and the second gradient image. ### Claim 6 (depends on 1) The method of claim 1, wherein the tooth template matching technique comprises: consecutively positioning a tooth template image at a plurality of locations in the difference image; and determining a plurality of correlation coefficients corresponding to the plurality of locations in the difference image based on the tooth template image and a plurality of pixels of the difference image. ### Claim 7 (depends on 1) The method of claim 1, wherein determining the tooth line comprises generating a plurality of candidate tooth lines based on a template tooth line associated with the selected bucket template image, wherein each of the plurality of candidate tooth lines are scaled and rotated version of the template tooth line. ### Claim 8 (depends on 7) The method of claim 7, wherein determining the tooth line comprises determining a fit score value as a sum of pixel values of the response map overlapping with the determined tooth line. ### Claim 9 (depends on 1) The method of claim 1, wherein determining the tooth condition comprises: determining a plurality of pixels in the difference image around a position of a tooth of a tooth line; determining a missing score value as a sum of the plurality of pixels of the difference image; and comparing the missing score value with a missing measurement threshold value and setting a missing flag to a value of one if the sum exceeds the missing measurement threshold value. ### Claim 10 (depends on 9) The method of claim 9, wherein determining the tooth condition comprises: initiating a missing count value to zero; determining a plurality of missing flags corresponding to a plurality of image frames; adding the plurality of missing flags to the missing count; and determining a tooth condition when the missing count is greater than a missing count threshold value. ### Claim 11 (depends on 1) The method of claim 1, wherein receiving the pair of image frames comprises rejecting an image frame based on the difference image. ### Claim 12 A system, comprising: an imaging device fitted on a rope mine shovel disposed on a rope mine shovel configured to carry a mining load, configured to acquire a plurality of image frames; a processing device communicatively coupled to the imaging device and configured to: receive a pair of image frames wherein the pair of image frames comprises a first image frame and a second image frame; detect a corresponding tooth line region for the pair of image frames to generate a pair of tooth line regions based on a shovel template set, wherein the shovel template set comprises a plurality of bucket template images, a plurality of tooth line images, a plurality of locations corresponding to the plurality of bucket template images, and a plurality of locations of tooth line images, and wherein the pair of tooth line regions comprises a first tooth line region in the first image and a second tooth line region in the second image; determine a difference image based on the pair of image frames and the pair of tooth line regions; determine a response map representative of possible tooth positions based on the difference image using a tooth template matching technique; select a tooth line among a plurality of candidate tooth lines based on the response map; determine a tooth condition based on the tooth line and the difference image; and notify the tooth condition to an operator of the rope mine shovel. ### Claim 13 (depends on 12) The system of claim 12, wherein the processing device is further configured to emit an alarm, or stop the rope mine shovel, or both. ### Claim 14 (depends on 12) The system of claim 12, wherein the processing device is further configured to: generate a plurality of image tiles corresponding to an image frame among the pair of image frames; determine a correlation between a bucket template image among a plurality of bucket template images and an image tile among the plurality of image tiles to generate a plurality of matching scores; select a bucket template image and an associated image tile corresponding to a minimum value among the plurality of matching scores; and determine a location of the tooth line region based on location of the selected image tile, location of the selected bucket template image, and a location of tooth line region associated with the selected bucket template image. ### Claim 15 (depends on 12) The system of claim 12, wherein the processing device is further configured to: determine a homography transformation of the first tooth line image with respect to the second tooth line image; and generate a transformed image by applying the homography transformation to the first image. ### Claim 16 (depends on 15) The system of claim 15, wherein the processing device is further configured to: determine a first gradient image corresponding to the transformed image and a second gradient image corresponding to the second image frame; and determine a difference between the first gradient image and the second gradient image. ### Claim 17 (depends on 12) The system of claim 12, wherein the processing device is further configured to: consecutively position a tooth template image at a plurality of locations on the difference image; and determine a plurality of correlation coefficients corresponding to the plurality of locations in the difference image based on the tooth template image and a plurality of pixels of the difference image. ### Claim 18 (depends on 12) The system of claim 12, wherein the processing device is further configured to generate a plurality of candidate tooth lines based on a template tooth line associated with the selected bucket template image, wherein each of the plurality of candidate tooth lines are scaled and rotated version of the template tooth line. ### Claim 19 (depends on 18) The system of claim 18, wherein the processing device is further configured to determine a fit score value as a sum of pixel values of the response map overlapping with a candidate tooth line among the plurality of tooth lines. ### Claim 20 (depends on 12) The system of claim 12, wherein the processing device is further configured to: determine a plurality of pixels in the difference image around a position of a tooth of a tooth line; determine a missing score value as a sum of the plurality of pixels of the difference image; and compare the missing score value with a missing measurement threshold value and setting a missing flag to a value of one if the sum exceeds the missing measurement threshold value. ### Claim 21 (depends on 12) The system of claim 12, wherein the processing device is further configured to: initiate a missing count value to zero; determine a plurality of missing flags corresponding to a plurality of image frames; add the plurality of missing flags to the missing count; and determine a tooth condition when the missing count is greater than a missing count threshold value. ### Claim 22 (depends on 12) The system of claim 12, wherein the processing device is configured to reject an image frame based on the difference image. ### Claim 23 A method, comprising: receiving a pair of image frames from a camera disposed on a rope mine shovel configured to carry a mining load, wherein the pair of image frames comprises a first image frame and a second image frame; detecting a corresponding metal part region for the pair of image frames to generate a pair of metal part regions based on a shovel template set, wherein the shovel template set comprises a plurality of shovel template images, a plurality of metal part images, a plurality of locations corresponding to the plurality of shovel template images, and a plurality of locations of metal part images, and wherein the pair of metal part regions comprises a first metal part region in the first image and a second metal part region in the second image; determining a difference image based on the pair of image frames and the pair of metal part regions; determining a response map representative of possible metal part positions based on the difference image using a metal part template matching technique; determining a metal part condition based on the response map and the difference image and notifying the metal part condition to an operator of the rope mine shovel.