# US 11,442,464 B2 · Bird's eye view map based recognition and motion prediction for autonomous systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,464 B2 - Document ID: US11442464B2 - Status: Granted - Kind: B2 - Application number: 16/829,045 - Filed: 2020-03-25 - Published: 2021-09-30 - Granted: 2022-09-13 - Earliest claimed: 2020-03-25 - Assignees: Mitsubishi Electric Research Laboratories, Inc. - Applicants: Mitsubishi Electric Research Laboratories, Inc. - Inventors: Siheng Chen; Pengxiang Wu - Web page: https://patentgrep.com/patent/US11442464B2 ## CPC - G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)} - B60W 60/0027: {using trajectory prediction for other traffic participants} - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2420/408: Radar; Laser, e.g. lidar - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30241: Trajectory - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A control system for controlling a motion of a vehicle is disclosed, where the control system comprises an input interface to accept measurements of the environment and an image processor to generate an extended bird's eye view (BEV) image from the measurements. The BEV image has a set of pixels carrying information about objects in the environment, and a pixel in the set of pixels is associated with a time sequence of future positions of the pixel in subsequent time steps representing a prediction of a future motion of an object. The control system further comprises a motion planner to produce a motion trajectory of the vehicle using the extended BEV image. The control system further comprises a controller to control an actuator of the vehicle based on the produced motion trajectory. ## Claims ### Claim 1 A control system for controlling a motion of a vehicle, comprising: a processor; and a memory having instructions stored thereon that, when executed by the processor, cause the control system to: submit the measurements to a multi-head neural network to generate an extended bird's eye view (BEV) image having a set of pixels carrying information about objects in the environment based on the measurements, wherein a pixel in the set of pixels is associated with a time sequence of future positions of the pixel in subsequent time steps representing a prediction of a future motion of an object, wherein the multi-head neural network includes a feature extraction network connected to a cell classification head, a motion prediction head, and a current motion state estimation head jointly trained with each other; produce a motion trajectory of the vehicle using the extended BEV image; and control an actuator of the vehicle based on the motion trajectory. ### Claim 2 (depends on 1) The control system of claim 1, wherein the cell classification head is trained to output a label of the object of the pixel, wherein the current motion state estimation head is trained to output whether the object is static or moving, and wherein the motion prediction head is trained to output a prediction of motion of the pixel corresponding to motion of the object. ### Claim 3 (depends on 2) The control system of claim 2, wherein the processor determines the time sequence of future positions of the pixel based on the outputs from the motion prediction head. ### Claim 4 (depends on 2) The control system of claim 2, wherein the processor ignores the output of the motion prediction head when the pixel is classified as a background, the object of the pixel is estimated as static, or combination thereof. ### Claim 5 (depends on 1) The control system of claim 1, wherein the feature extraction network is a spatio-temporal network configured to extract feature in both spatial and temporal dimensions. ### Claim 6 (depends on 1) The control system of claim 1, wherein the feature extraction network includes a spatio-temporal pyramid network (STPN) for feature extraction. ### Claim 7 (depends on 6) The control system of claim 6, wherein the STPN includes a sequence of spatio-temporal convolution (STC) blocks, each STC block enlarges information in channels of an input and shrinks space and temporal dimensions of the input forming a pyramid structure of the STPN. ### Claim 8 (depends on 7) The control system of claim 7, wherein the output of each STC block is processed with a temporal pooling that preserves a largest value across temporal dimension of the output of the STC block to shrink the temporal dimensions of the output into one. ### Claim 9 (depends on 8) The control system of claim 8, the sequence of STC blocks of the STPN iteratively process the input in a forward direction, such that the output of one STC block is an input to the next STC block, wherein the STPN combines the outputs of the STC blocks processed by the temporal pooling in a reverse direction, such that a combination of two STC outputs is combined with the output of the previous STC block. ### Claim 10 (depends on 1) The control system of claim 1, wherein each of the cell classification head, the motion prediction head, and the current motion state estimation head operate concurrently. ### Claim 11 (depends on 1) The control system of claim 1, wherein the measurements correspond to a sequence of three dimensional (3D) point clouds, and wherein the sequence of 3D point clouds is generated from one of a light detection and ranging (LiDAR) sensor or a radio detection and ranging (RADAR) sensor. ### Claim 12 (depends on 1) The control system of claim 1, wherein the pixel is classified with a label of the object occupying a location corresponding to a position of the pixel in the extended BEV image at a current time step. ### Claim 13 A method for controlling a motion of a vehicle, comprising: accepting measurements of an environment; generating, from the measurements, an extended bird's eye view (BEV) image having a set of pixels carrying an information about objects in the environment, wherein a pixel in the set of pixels is associated with a time sequence of future positions of the pixel in subsequent time steps representing a prediction of a future motion of an object, wherein the extended BEV image is generated via a multi-head neural network including a feature extraction network connected to a cell classification head, a motion prediction head, and a current motion state estimation head jointly trained with each other; and producing a motion trajectory of the vehicle using the extended BEV image; and controlling an actuator of the vehicle based on the motion trajectory. ### Claim 14 (depends on 13) The method of claim 13, wherein the pixel in the set of pixels is classified with a label of the object occupying a location corresponding to a position of the pixel in the extended BEV image at a current time step.