# US 2019/0171912 A1 · Multiple Stage Image Based Object Detection and Recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0171912 A1 - Document ID: US20190171912A1 - Status: Published application - Kind: A1 - Application number: 15/972,566 - Filed: 2018-05-07 - Published: 2019-06-06 - Earliest claimed: 2017-12-05 - Assignees: Uber Technologies, Inc. - Applicants: Uber Technologies, Inc. - Inventors: Carlos Vallespi-Gonzalez; Joseph Lawrence Amato; George Totolos, JR. - Web page: https://patentgrep.com/patent/US20190171912A1 ## CPC - G06V 20/64: Three-dimensional [3D] objects - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches - G06F 18/24323: {Tree-organised classifiers} - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06T 15/08: Volume rendering - G06V 10/28: Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns - G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis - G06V 10/56: relating to colour - G06V 10/764: using classification, e.g. of video objects - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/584: {of vehicle lights or traffic lights} - G05D 1/0238: {using obstacle or wall sensors (G05D1/0246 and G05D1/0289 take precedence; lidar systems designed for anti-collision purposes G01S17/93)} - G05D 1/628: following the obstacle profile, e.g. a wall or undulated terrain - G06T 2207/20081: Training; Learning - G06T 2207/30261: Obstacle - G06T 2210/12: Bounding box - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES ## Abstract Systems, methods, tangible non-transitory computer-readable media, and devices for autonomous vehicle operation are provided. For example, a computing system can receive object data that includes portions of sensor data. The computing system can determine, in a first stage of a multiple stage classification using hardware components, one or more first stage characteristics of the portions of sensor data based on a first machine-learned model. In a second stage of the multiple stage classification, the computing system can determine second stage characteristics of the portions of sensor data based on a second machine-learned model. The computing system can generate an object output based on the first stage characteristics and the second stage characteristics. The object output can include indications associated with detection of objects in the portions of sensor data. ## Claims ### Claim 1 A computer-implemented method of autonomous vehicle operation, the computer-implemented method comprising: receiving, by a computing system comprising one or more computing devices, object data comprising one or more portions of sensor data; determining, by the computing system, in a first stage of a multiple stage classification using one or more hardware components, one or more first stage characteristics of the one or more portions of sensor data based in part on a first machine-learned model; determining, by the computing system, in a second stage of the multiple stage classification, one or more second stage characteristics of the one or more portions of sensor data based in part on a second machine-learned model; and generating, by the computing system, an object output based in part on the one or more first stage characteristics and the one or more second stage characteristics, wherein the object output comprises one or more indications associated with detection of one or more objects in the one or more portions of sensor data. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the one or more hardware components comprise a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a graphics processing unit (GPU). ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the one or more portions of sensor data comprise one or more images and further comprising: generating, by the computing system, in the first stage and based in part on the object data, visual descriptor output associated with the one or more images, the visual descriptor output comprising color hue information, color saturation information, brightness information, or histogram of oriented gradients information, wherein the one or more first stage characteristics are determined based in part on the visual descriptor output. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, further comprising: generating, by the computing system, in the second stage and based in part on the visual descriptor output from the first stage, a heat map associated with the one or more images, the heat map comprising a plurality of areas associated with a probability of at least one of the one or more objects being within the respective one of the plurality of areas, wherein the one or more second stage characteristics are determined based in part on the heat map. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising: determining, by the computing system, based in part on the visual descriptor output from the first stage, one or more portions of the one or more images that are associated with one or more background portions, wherein the determining the one or more second stage characteristics in the second stage comprises determining the one or more second stage characteristics in the second stage and excluding the one or more portions of the one or more images that are associated with the one or more background portions. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the one or more first stage characteristics and the one or more second stage characteristics are based in part on traversal of the first machine-learned model and the second machine-learned model respectively, the first machine-learned model comprising a first plurality of nodes associated with a first plurality of classifier labels and the second machine-learned model comprising a second plurality of nodes associated with a second plurality of classifier labels. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, further comprising: determining, by the computing system, based in part on the object data and the second machine-learned model, an amount of false positive determinations of the one or more second stage characteristics of the one or more objects that has occurred; and terminating, by the computing system, the traversal of a second decision tree when the amount of the false positives determined to have occurred exceeds a predetermined threshold level. ### Claim 8 (depends on 6) The computer-implemented method of claim 6, wherein at least one node of the second plurality of nodes in the second machine-learned model is a terminal node of the first plurality of nodes in the first machine-learned model, the second machine-learned model comprises an equal number of nodes as the first plurality of nodes, or the second machine-learned model comprises a greater number of nodes than the first plurality of nodes. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, based in part on the object output, locations for one or more bounding shapes associated with the one or more objects in the one or more portions of sensor data; selecting, by the computing system, based in part on an image processing technique comprising non-maximum suppression, a set of the locations for the one or more bounding shapes; and generating, by the computing system, the one or more bounding shapes in the set of the locations for the one or more bounding shapes. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the one or more first stage characteristics comprise an indication of when a portion of the one or more portions of sensor data is foreground or background and the one or more second stage characteristics comprise an object classification of a foreground portion of the one or more portions of sensor data. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the one or more portions of sensor data are based in part on sensor output from one or more sensors comprising one or more light detection and ranging devices (LIDAR), one or more cameras, one or more radar devices, one or more sonar devices, or one or more thermal imaging devices. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the first machine-learned model or the second machine-learned model is based in part on one or more classification techniques comprising a neural network, a random forest classifier, gradient boosting, a support vector machine, a logistic regression classifier, or a boosted forest classifier. ### Claim 13 One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising: receiving object data comprising one or more portions of sensor data; determining, in a first stage of a multiple stage classification, one or more first stage characteristics of the one or more portions of sensor data based in part on a first machine-learned model; determining, in a second stage of the multiple stage classification, one or more second stage characteristics of the one or more portions of sensor data based in part on a second machine-learned model; and generating, an object output based in part on the one or more first stage characteristics and the one or more second stage characteristics, wherein the object output comprises one or more indications associated with detection of one or more objects in the one or more portions of sensor data. ### Claim 14 (depends on 13) The one or more tangible, non-transitory computer-readable media of claim 13, wherein the one or more portions of sensor data comprise one or more images and further comprising: generating, in the first stage and based in part on the object data, visual descriptor output associated with the one or more images, the visual descriptor output comprising color hue information, color saturation information, brightness information, or histogram of oriented gradients information, wherein the one or more first stage characteristics are determined based in part on the visual descriptor output. ### Claim 15 (depends on 13) The one or more tangible, non-transitory computer-readable media of claim 13, further comprising: determining, based in part on the object output, locations for one or more bounding shapes associated with the one or more objects in the one or more portions of sensor data; selecting, based in part on an image processing technique comprising non-maximum suppression, a set of the locations for the one or more bounding shapes; and generating the one or more bounding shapes in the set of the locations for the one or more bounding shapes. ### Claim 16 A computing system comprising: one or more processors; and a memory comprising one or more computer-readable media, the memory storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising: receiving object data comprising one or more portions of sensor data; determining, in a first stage of a multiple stage classification using one or more hardware components, one or more first stage characteristics of the one or more portions of sensor data based in part on a first machine-learned model; determining, in a second stage of the multiple stage classification, one or more second stage characteristics of the one or more portions of sensor data based in part on a second machine-learned model; and generating, an object output based in part on the one or more first stage characteristics and the one or more second stage characteristics, wherein the object output comprises one or more indications associated with detection of one or more objects in the one or more portions of sensor data. ### Claim 17 (depends on 16) The computing system of claim 16, wherein the one or more hardware components comprise a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a graphics processing unit (GPU). ### Claim 18 (depends on 16) The computing system of claim 16, wherein the one or more portions of sensor data comprise one or more images and further comprising: generating, in the first stage and based in part on the object data, visual descriptor output associated with the one or more images, the visual descriptor output comprising color hue information, color saturation information, brightness information, or histogram of oriented gradients information, wherein the one or more first stage characteristics are determined based in part on the visual descriptor output. ### Claim 19 (depends on 16) The computing system of claim 16, wherein the one or more first stage characteristics and the one or more second stage characteristics are based in part on traversal of a first decision tree of the first machine-learned model and a second decision tree of the second machine-learned model respectively, the first decision tree comprising a first plurality of nodes associated with a first plurality of classifier labels and the second decision tree based in part on the first decision tree and comprising a second plurality of nodes associated with a second plurality of classifier labels. ### Claim 20 (depends on 16) The computing system of claim 16, further comprising: determining, based in part on the object output, locations for one or more bounding shapes associated with the one or more objects in the one or more portions of sensor data; selecting, based in part on an image processing technique comprising non-maximum suppression, a set of the locations for the one or more bounding shapes; and generating the one or more bounding shapes in the set of the locations for the one or more bounding shapes.