# US 12,183,064 B2 · Deep learning inference systems and methods for imaging systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,064 B2 - Document ID: US12183064B2 - Status: Granted - Kind: B2 - Application number: 17/224,040 - Filed: 2021-04-06 - Published: 2021-07-22 - Granted: 2024-12-31 - Earliest claimed: 2018-10-15 - Assignees: Teledyne FLIR Commercial Systems, Inc. - Applicants: Teledyne FLIR Commercial Systems, Inc. - Inventors: Shuen Yan Stephen Se; Damian Michael Patrick Nesbitt; Ian Kenneth Stadler - Web page: https://patentgrep.com/patent/US12183064B2 ## CPC - G06V 10/82: using neural networks - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 5/04: Inference or reasoning models - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries - G06V 10/764: using classification, e.g. of video objects - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - H04N 23/61: based on recognised objects - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Deep learning inference for imaging systems includes an imaging device comprising an image capture component and a vision processing unit configured to process the image through a first trained inference network to determine a first inference result. A host system trains a neural network for image classification and produces and transmits the first trained inference network to the vision processing unit of the imaging device. The first trained inference network provides stand-alone image classification, an object detection and/or a confidence score to the imaging device. The vision processing unit may further comprise a two or more trained inference networks configured to receive an inference result as an input and output a second inference result. A stereo camera system uses features and/or results from one or more inference network to control processing of three-dimensional data for the detected object. ## Claims ### Claim 1 A system comprising: two or more image capture components configured to capture a corresponding pair of two-dimensional images of a scene; a vision processing unit configured to process the pair of two-dimensional images through a first trained inference network to detect an object of interest and determine a first inference result comprising a bounding box corresponding to a location of the detected object in at least one of the images, a classification of the detected object, and a confidence score associated with the classification, wherein at least one image feature computed during operation of the first trained inference network is used as a feature for stereo matching; a processing component configured to process the first inference result and determine real-world coordinates of the detected object based on the location of the bounding box in the pair of two-dimensional images; and a second trained inference network, wherein: if the confidence score exceeds the threshold, the second trained inference network is configured to receive the first inference result as an input and output a second inference result; wherein three-dimensional data is used to control a robotic system to interact with the detected object; wherein the vision processing unit is further configured to perform stereo matching on regions of interest in the two-dimensional images based on the bounding box to produce spatial location information for the object of interest; wherein the stereo matching is performed when the object of interest is detected; and wherein the stereo matching includes a correlation-based method and/or a feature-based method to generate a three-dimensional point cloud. ### Claim 2 (depends on 1) The system of claim 1, wherein the bounding box comprises a two-dimensional and/or three-dimensional location. ### Claim 3 (depends on 1) The system of claim 1, wherein the two or more image capture components comprise a binocular stereo camera system; and wherein the two or more image capture components are calibrated and produce rectified image pairs for input to the first trained inference network. ### Claim 4 A method comprising: loading onto a vision processing unit of an imaging device a first trained inference network; capturing an image pair of a scene using two or more imaging components of the imaging device; processing the image pair through the first trained inference network to detect an object of interest and determine a first inference result comprising bounding boxes corresponding to two-dimensional locations in at least one of the images in the image pair of the detected object in at least one of the images, wherein at least one image feature computed during operation of the first trained inference network are used as features for stereo matching when objects of interest are detected; and determining, based on the first inference result, an action for the image pair; and controlling a robotic system to interact with at least one of the detected objects based on three dimensional data; performing stereo matching on regions of interest in the image pair based on the bounding boxes to produce spatial location information for the objects of interest, wherein the stereo matching is performed when objects of interest are detected; and wherein the stereo matching includes a correlation-based method and/or a feature-based method to generate a three-dimensional point cloud. ### Claim 5 (depends on 4) The method of claim 4, further comprising: training, by a host system, a neural network for image classification to produce the first trained inference network; optimizing the first trained inference network for an image classification application; converting the optimized first trained inference network for operation on the vision processing unit of the imaging device; and uploading the converted optimized first trained inference network from the host system to the vision processing unit of the imaging device. ### Claim 6 (depends on 4) The method of claim 4, wherein the first inference result further includes an anomaly detection. ### Claim 7 (depends on 4) The method of claim 4, further comprising transmitting the first inference result to a host system. ### Claim 8 (depends on 4) The method of claim 4, wherein the action includes streaming the image pair to a host system, storing the image pair on the imaging device, and/or communicating the inference result to a peripheral device via GPIO. ### Claim 9 (depends on 4) The method of claim 4, wherein loading onto the vision processing unit further comprises encrypting and storing the first trained inference network in a first memory; and wherein the method further comprises decrypting and loading the first trained inference network into a random-access memory for inference; and wherein the image pair comprises a visible spectrum image, infrared image, color image, hyperspectral image, grayscale image, monochrome image and/or a time-of-flight image. ### Claim 10 A system comprising: two or more image capture components configured to capture a pair of images of a scene; a vision processing unit configured to process the image pair through a first trained inference network to determine a first inference result, wherein the first trained inference network detects objects of interest in the image pair and generates bounding boxes of corresponding two-dimensional locations in at least one of the images in the image pair, wherein at least one image feature computed during operation of the first trained inference network are used as features for stereo matching when objects of interest are detected; and a processing component configured to process the first inference result and determine, based on the first inference result, an action for the image pair; wherein three-dimensional data is used to control a robotic system to interact with at least one of the detected objects; wherein the vision processing unit is further configured to perform stereo matching on regions of interest in the image pair based on the bounding boxes to produce spatial location information for the objects of interest; wherein the stereo matching is performed when objects of interest are detected; and wherein the stereo matching includes a correlation-based method and/or a feature-based method to generate a three-dimensional point cloud. ### Claim 11 (depends on 10) The system of claim 10, further comprising: a host system configured to train a neural network for image classification and produce the first trained inference network and transmit the first trained inference network to the vision processing unit; and wherein the host system is further configured to optimize the first trained inference network for an image classification application, convert the optimized first trained inference network for operation on the vision processing unit, and upload the converted first trained inference network from the host system to the vision processing unit. ### Claim 12 (depends on 10) The system of claim 10, wherein the first inference result further includes an anomaly detection, wherein the anomaly detection comprises identifying defects and indicating where the defects are in the detected object. ### Claim 13 (depends on 10) The system of claim 10, further comprises an image processing component configured to receive the image pair captured by the two or more image capture components and modify the image pair for input to the first trained inference network. ### Claim 14 (depends on 10) The system of claim 10, wherein the processing component is further configured to stream the image pair to a host system, store the image pair on the two or more image capture components, and/or communicate the inference result to a peripheral device via GPIO. ### Claim 15 (depends on 10) The system of claim 10, wherein the two or more image capture components are configured to capture visible spectrum images, infrared images, color images, hyperspectral images, grayscale images, monochrome images and/or time-of-flight images. ### Claim 16 (depends on 10) The system of claim 10, wherein the first trained inference network is encrypted and stored in a first memory, and wherein the first trained inference network is decrypted and loaded into a random-access memory for inference. ### Claim 17 (depends on 10) The system of claim 10, wherein the processing component is further configured to determine real-world coordinates of the detected objects based on the locations of the bounding boxes in the image pair.