# US 11,443,148 B2 · Multiple stage image based object detection and recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,148 B2 - Document ID: US11443148B2 - Status: Granted - Kind: B2 - Application number: 17/007,969 - Filed: 2020-08-31 - Published: 2020-12-17 - Granted: 2022-09-13 - Earliest claimed: 2017-12-05 - Assignees: Uatc, LLC - Applicants: Uatc, LLC - Inventors: Carlos Vallespi-Gonzalez; Joseph Lawrence Amato; George Totolos, Jr. - Web page: https://patentgrep.com/patent/US11443148B2 ## CPC - G06V 20/64: Three-dimensional [3D] objects - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches - G06F 18/24323: {Tree-organised classifiers} - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06T 15/08: Volume rendering - G06V 10/28: Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns - G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis - G06V 10/56: relating to colour - G06V 10/764: using classification, e.g. of video objects - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/584: {of vehicle lights or traffic lights} - G05D 1/0238: {using obstacle or wall sensors (G05D1/0246 and G05D1/0289 take precedence; lidar systems designed for anti-collision purposes G01S17/93)} - G05D 1/628: following the obstacle profile, e.g. a wall or undulated terrain - G06T 2207/20081: Training; Learning - G06T 2207/30261: Obstacle - G06T 2210/12: Bounding box - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES ## Abstract Systems, methods, tangible non-transitory computer-readable media, and devices for autonomous vehicle operation are provided. For example, a computing system can receive object data that includes portions of sensor data. The computing system can determine, in a first stage of a multiple stage classification using hardware components, one or more first stage characteristics of the portions of sensor data based on a first machine-learned model. In a second stage of the multiple stage classification, the computing system can determine second stage characteristics of the portions of sensor data based on a second machine-learned model. The computing system can generate an object output based on the first stage characteristics and the second stage characteristics. The object output can include indications associated with detection of objects in the portions of sensor data. ## Claims ### Claim 1 A computer-implemented method of autonomous vehicle operation, the computer-implemented method comprising: receiving sensor data comprising image; determining in a first stage of a multiple stage classification using one or more hardware components, one or more first stage characteristics of the sensor data based in part on a first machine-learned model, wherein the first stage characteristics are determined by the first machine-learned model with a first level of confidence; excluding a portion of the sensor data corresponding to one or more areas within the image from an input to a second stage of the multiple stage classification based on the one or more first stage characteristics of the sensor data; determining in the second stage of the multiple stage classification and based on the input to the second stage, one or more second stage characteristics of the sensor data based in part on a second machine-learned model, wherein the second stage characteristics are determined by the second machine-learned model with a second level of confidence that is higher than the first level of confidence; and generating an object output based on the second stage characteristics, wherein the object output describes detection of one or more objects in the sensor data. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the one or more first stage characteristics of the sensor data determined in the first stage of the multiple stage classification describes a likelihood that the portion of the sensor data that is excluded from the input to the second stage of the multiple stage classification contains objects. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising generating in the first stage, a heat map associated with the sensor data, the heat map describing a probability of an object being contained within a respective area of a plurality of areas of the sensor data. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein excluding the portion of the sensor data from the input to the second stage of the multiple stage classification based on the one or more first stage characteristics of the sensor data comprises excluding the portion of the sensor data based on the heat map. ### Claim 5 (depends on 3) The computer-implemented method of claim 3, wherein excluding the portion of the sensor data from the input to the second stage of the multiple stage classification based on the one or more first stage characteristics of the sensor data comprises excluding the portion of the sensor data based on the visual descriptor output associated with the sensor data. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the portion of the sensor data that is excluded from the input to the second stage of the multiple stage classification is associated with one or more background portions of the sensor data. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the input to the second stage of the multiple stage classification is associated with one or more foreground portions of the sensor data. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: generating in the first stage and based in part on the sensor data, visual descriptor output associated with the sensor data, the visual descriptor output comprising color hue information, color saturation information, brightness information, or histogram of oriented gradients information, wherein the one or more first stage characteristics are determined based in part on the visual descriptor output. ### Claim 9 A computing system comprising: one or more processors; and a memory comprising one or more computer-readable media, the memory storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising: receiving sensor data comprising an image; determining, in a first stage of a multiple stage classification using one or more hardware components, one or more first stage characteristics of the sensor data based in part on a first machine-learned model, wherein the first stage characteristics are determined by the first machine-learned model with a first level of confidence; excluding a portion of the sensor data corresponding to one or more areas within the image from an input to a second stage of the multiple stage classification based on the one or more first stage characteristics of the sensor data, determining in the second stage of the multiple stage classification and based on the input to the second stage, one or more second stage characteristics of the sensor data based in part on a second machine-learned model, wherein the second stage characteristics are determined by the second machine-learned model with a second level of confidence that is higher than the first level of confidence; and generating an object output based on the second stage characteristics, wherein the object output describes detection of one or more objects in the sensor data. ### Claim 10 (depends on 9) The computing system of claim 9, wherein the one or more first stage characteristics of the sensor data determined in the first stage of the multiple stage classification describes a likelihood that the portion of the sensor data that is excluded from the input to the second stage of the multiple stage classification contains objects. ### Claim 11 (depends on 9) The computing system of claim 9, wherein the operations further comprise generating, in the first stage, a heat map associated with the sensor data, the heat map describing a probability of an object being contained within a respective area of a plurality of areas of the sensor data. ### Claim 12 (depends on 9) The computing system of claim 9, wherein the portion of the sensor data that is excluded from the input to the second stage of the multiple stage classification is associated with one or more background portions of the sensor data. ### Claim 13 (depends on 9) The computing system of claim 9, wherein the input to the second stage of the multiple stage classification is associated with one or more foreground portions of the sensor data. ### Claim 14 (depends on 9) The computing system of claim 9, wherein the operations further comprise generating, in the first stage and based in part on the sensor data, visual descriptor output associated with the sensor data, the visual descriptor output comprising color hue information, color saturation information, brightness information, or histogram of oriented gradients information, wherein the one or more first stage characteristics are determined based in part on the visual descriptor output. ### Claim 15 (depends on 14) The computing system of claim 14, wherein excluding the portion of the sensor data from the input to the second stage of the multiple stage classification based on the one or more first stage characteristics of the sensor data comprises excluding the portion of the sensor data based on the visual descriptor output associated with the sensor data. ### Claim 16 One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising: receiving, by a computing system comprising one or more computing devices, sensor data comprising image; determining, by the computing system, in a first stage of a multiple stage classification using one or more hardware components, one or more first stage characteristics of the sensor data based in part on a first machine-learned model, wherein the first stage characteristics are determined by the first machine-learned model with a first level of confidence; excluding, by the computing system, a portion of the sensor data corresponding to one or more areas within the image from an input to a second stage of the multiple stage classification based on the one or more first stage characteristics of the sensor data; determining, by the computing system, in the second stage of the multiple stage classification and based on the input to the second stage, one or more second stage characteristics of the sensor data based in part on a second machine-learned model, wherein the second stage characteristics are determined by the second machine-learned model with a second level of confidence that is higher than the first level of confidence; and generating, by the computing system, an object output based on the second stage characteristics, wherein the object output describes detection of one or more objects in the sensor data. ### Claim 17 (depends on 16) One or more tangible, non-transitory computer-readable media of claim 16, wherein the one or more first stage characteristics of the sensor data determined in the first stage of the multiple stage classifications describes a likelihood that the portion of the sensor data that is excluded from the input to the second stage of the multiple stage classification contains objects. ### Claim 18 (depends on 16) The one or more tangible, non-transitory computer-readable media of claim 16, wherein the operations further comprise generating, by the computing system, in the first stage, a heat map associated with the sensor data, the heat map describing a probability of an object being contained within a respective area of a plurality of areas of the sensor data. ### Claim 19 (depends on 18) The one or more tangible, non-transitory computer-readable media of claim 18, wherein excluding the portion of the sensor data from the input to the second stage of the multiple stage classification based on the one or more first stage characteristics of the sensor data comprises excluding the portion of the sensor data based on the heat map.