# US 10,482,331 B2 · Stixel estimation methods and systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,331 B2 - Document ID: US10482331B2 - Status: Granted - Kind: B2 - Application number: 15/353,359 - Filed: 2016-11-16 - Published: 2017-05-25 - Granted: 2019-11-19 - Earliest claimed: 2015-11-20 - Assignees: GM Global Technology Operations LLC - Applicants: GM Global Technology Operations LLC - Inventors: Shuqing Zeng - Web page: https://patentgrep.com/patent/US10482331B2 ## CPC - G06V 10/82: using neural networks - G01S 13/867: {Combination of radar systems with cameras} - G01S 13/931: of land vehicles - G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]} - G06T 7/77: using statistical methods - G06V 10/764: using classification, e.g. of video objects - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06T 2207/10004: Still image; Photographic image - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10044: Radar image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20104: Interactive definition of region of interest [ROI] - G06T 2207/30261: Obstacle - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Methods and systems are provided for detecting an object. In one embodiment, a method includes: receiving, by a processor, image data from an image sensor; receiving, by a processor, radar data from a radar system; processing, by the processor, the image data from the image sensor and the radar data from the radar system using a deep learning method; and detecting, by the processor, an object based on the processing. ## Claims ### Claim 1 A method for detecting an object, comprising: receiving, by a processor, image data from an image sensor; receiving, by the processor, radar data from a radar system; processing, by the processor, the image data from the image sensor to determine one or more stixels; processing, by the processor, the radar data from the radar system to determine one or more presence vectors; fusing, by the processor, the one or more stixels and the one or more presence vectors using a deep learning method; estimating, by the processor, a motion of the one or more stixels based on the fused stixels and presence vectors; and detecting, by the processor, an object based on the estimated motion of the one or more stixels. ### Claim 2 (depends on 1) The method of claim 1, wherein the processing comprises: forming a region of interest (ROI) window of an image based on the image data; and moving the ROI window to a plurality of locations, from a left side of the image location to a right side of the image, to sequentially determine a presence of a stixel at each location. ### Claim 3 (depends on 1) The method of claim 1, wherein the processing comprises: forming a region of interest (ROI) window based on the image data; determining whether a centerline of the ROI window includes a stixel using a convolution neural network; and reporting a probability, a location, a height, and a class label when a stixel is determined. ### Claim 4 (depends on 1) The method of claim 1, further comprising determining a velocity for each of the fused stixels and presence vectors, and wherein the detecting the object is based on the velocity of each of the fused stixels and presence vectors. ### Claim 5 (depends on 4) The method of claim 4, further comprising determining a displacement for each of the fused stixels and presence vectors, and wherein the detecting the object is based on the displacement of each of the fused stixels and presence vectors. ### Claim 6 A system for detecting an object, comprising: an image sensor that generates image data; a radar system that generates radar data; and a computer module that, by a processor, processes the image data from the image sensor to determine one or more stixels, processes, the radar data from the radar system to determine one or more presence vectors, fuses the one or more stixels and the one or more presence vectors, estimates, a motion of the one or more stixels based on the fused stixels and presence vectors; and and detects an object based on the estimated motion of the one or more stixels. ### Claim 7 (depends on 6) The system of claim 6, wherein the computer module forms a region of interest (ROI) window of an image based on the image data; and moves the ROI window to a plurality of locations, from a left side of the image location to a right side of the image, to sequentially determine a presence of a stixel at each location. ### Claim 8 (depends on 6) The system of claim 6, wherein the computer module forms a region of interest (ROI) window based on the image data, determines whether a centerline of the ROI window includes a stixel using a convolution neural network, and reports a probability, a location, a height, and a class label when a stixel is determined. ### Claim 9 (depends on 6) The system of claim 6, wherein the computer module determines a velocity for each of the fused stixels and presence vectors, and detects the object based on the velocity of each of the fused stixels and presence vectors. ### Claim 10 (depends on 9) The system of claim 9, wherein the computer module determines a displacement for each of the fused stixels and presence vectors, and detects the object based on the displacement of each of the fused stixels and presence vectors. ### Claim 11 (depends on 6) The system of claim 6, wherein the image sensor and the radar system are associated with a vehicle, and wherein the control module detects the object in proximity to the vehicle. ### Claim 12 (depends on 1) The method of claim 1, wherein the estimating the motion is based on an optimal column displacement in pixel using a standard dynamic programming minimization method. ### Claim 13 (depends on 12) The method of claim 12, wherein the standard dynamic programming minimization method is a two-phase method with columns including locations of valid stixels of the one or more stixels and rows including motion.