# US 12,118,445 B1 · Temporal-based deformable kernels > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,445 B1 - Document ID: US12118445B1 - Status: Granted - Kind: B1 - Application number: 16/219,713 - Filed: 2018-12-13 - Granted: 2024-10-15 - Earliest claimed: 2018-12-13 - Assignees: Zoox, Inc. - Applicants: Zoox, Inc. - Inventors: Sarah Tariq - Web page: https://patentgrep.com/patent/US12118445B1 ## CPC - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - B60K 31/0008: {including means for detecting potential obstacles in vehicle path} - G06N 3/045: Combinations of networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06T 7/11: Region-based segmentation - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/254: involving subtraction of images - G06T 7/269: using gradient-based methods - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 10/243: {by compensating for image skew or non-uniform image deformations} - G06V 10/36: Applying a local operator, i.e. means to operate on image points situated in the vicinity of a given point; Non-linear local filtering operations, e.g. median filtering - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) - G06V 10/755: {Deformable models or variational models, e.g. snakes or active contours} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60K 2031/0016: {Identification of obstacles; Selection of a target vehicle} - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques are disclosed for implementing a convolutional neural network that determines an offset field for deforming a kernel to be used in a convolution. The offset field is temporally-based, at least in part, on data generated at an earlier time. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn shapes or configurations of such deformed kernels. The temporal-based deformable convolutions may be used for object identification, object matching, object classification, segmentation, and/or object tracking, in various examples. ## Claims ### Claim 1 A system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising: training a machine learned (ML) model to detect a first object in an environment, the training comprising: inputting a first image generated at a first time; inputting a second image generated at a second time different from the first time; determining, based at least in part on the first image and the second image, a tracking velocity associated with a second object, wherein the second object is represented in the first image and the second image, and an optical flow, wherein the tracking velocity and the optical flow are determined on a per-pixel basis between the first image and the second image; determining an offset field based at least in part on the tracking velocity and the optical flow; applying the offset field to a kernel to produce a modified kernel, wherein the kernel is configured to perform two-dimensional convolution and is characterized by a plurality of kernel elements, and applying the offset field comprises displacing an individual kernel element of the plurality of kernel elements in a horizontal, vertical, or diagonal direction; further wherein the modified kernel is associated with a pixel of the first image or a pixel of the second image; determining, on a per-pixel basis, an auxiliary loss based at least in part on an average offset vector of the offset field and a vector associated with the tracking velocity; determining that the auxiliary loss meets or exceeds a threshold; updating, based at least in part on the auxiliary loss meeting or exceeding the threshold, the ML model to generate an updated ML model; convolving, on a per-pixel basis, the second image with the modified kernel to produce an output; and transmitting the updated ML model to an autonomous vehicle, wherein detecting an object comprises outputting, by the updated ML model, one or more of a bounding box or an instance segmentation. ### Claim 2 (depends on 1) The system as claim 1 recites, the training further comprising determining that a number of training iterations has been performed. ### Claim 3 (depends on 2) The system as claim 2 recites, the training further comprising determining that a difference between a first auxiliary loss associated with a first training iteration and a second auxiliary loss associated with a second training iteration meets or exceeds a threshold. ### Claim 4 (depends on 1) The system as claim 1 recites, the training further comprising: downsampling the optical flow to determine a downsampled optical flow, wherein the auxiliary loss is based at least in part on the downsampled optical flow. ### Claim 5 (depends on 1) The system as claim 1 recites, wherein the tracking velocity associated with the second object represented in the first image and the second image is represented as an image embedding comprising a vertical component and a horizontal component. ### Claim 6 (depends on 1) The system as claim 1 recites, wherein prior to inputting the first image or the second image, one or more of the first image and the second image is retrieved from a memory device wherein it has been stored since it was generated. ### Claim 7 (depends on 1) The system as claim 1 recites, the training further comprising: generating a loss associated with tracking a bounding box corresponding to the second object between the first time and the second time, wherein generating the updated ML model is further based at least in part on the loss. ### Claim 8 (depends on 1) The system as claim 1 recites, wherein displacing the individual kernel element of the plurality of kernel elements is partially based on a shape of the second object or a shape of a bounding box. ### Claim 9 (depends on 1) The system of claim 1, wherein updating the machine learned model comprises varying weights associated with one or more parameters of the machine learned model. ### Claim 10 A computer-implemented method comprising: inputting first data into a machine learning model, the machine learning model trained at least in part by determining an auxiliary loss based at least in part on an average offset vector of an offset field, to one or more of track an object, detect the object, classify the object, or segment data associated with the object in an environment by performing operations comprising: inputting second data into the machine learning model; generating, for at least a portion of the first data, the offset field, wherein the offset field is based on a difference between the first data and the second data; applying the offset field to a kernel; generating, based on applying the offset field to the kernel, a modified kernel, and convolving, on a per-pixel basis, the portion of the first data with the modified kernel to produce an output feature map. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, the method further comprising controlling an autonomous vehicle based at least in part on the operations. ### Claim 12 (depends on 10) The computer-implemented method of claim 10, wherein the difference between the first data and the second data is based at least in part on a tracking velocity associated with the object, wherein the object is represented in the first data and the second data. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein the first data corresponds to a first image generated at a first time and the second data corresponds to a second image generated at a second time that is different from the first time. ### Claim 14 (depends on 10) The computer-implemented method of claim 10, wherein the difference is based at least in part on a first feature map associated with the first data and a second feature map associated with the second data, wherein the first feature map is output from a subnetwork in response to receiving the first data, and wherein the second feature map is output from the subnetwork in response to receiving the second data. ### Claim 15 (depends on 10) The computer-implemented method of claim 10, wherein the first data corresponds to the object at a first location in a first image and the second data corresponds to the object at a second location in a second image, wherein the difference between the first data and the second data is based at least in part on a difference between the first location and the second location. ### Claim 16 (depends on 10) The computer-implemented method of claim 10, wherein the difference between the first data and the second data is based at least in part on tracking a bounding box associated with the object or an optical flow associated with the first data and the second data. ### Claim 17 One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising: inputting first data into a machine learning model trained at least in part by determining an auxiliary loss based at least in part on an average offset vector of an offset field, to one or more of track an object, detect the object, classify the object, or segment data associated with the object in an environment by performing operations comprising: generating, for at least a portion of the first data, the offset field, wherein the offset field is based on a difference between the first data and second data; applying the offset field to a kernel; generating, based on applying the offset field to the kernel, a modified kernel; and convolving, on a per-pixel basis, the portion of the first data with the modified kernel to produce an output feature map. ### Claim 18 (depends on 17) The one or more non-transitory computer-readable media of claim 17, wherein the difference comprises an optical flow computed between a first feature map and a second feature map, wherein the first feature map is output by a subnetwork in response to receiving the first data, wherein the second feature map is output by the subnetwork in response to receiving the second data, wherein the first feature map is computed at a first time and the second feature map is computed at a second time prior to the first time and stored. ### Claim 19 (depends on 17) The one or more non-transitory computer-readable media of claim 17, the operations further comprising: capturing, at a first time, a first image including the object; and capturing, at a second time different than the first time, a second image including the object; wherein the first data and the second data both correspond, respectively, to the object in the first image and in the second image. ### Claim 20 (depends on 19) The one or more non-transitory computer-readable media of claim 19, wherein the difference is based at least in part on an optical flow determined based on the first image and the second image or a velocity of the object from the first image to the second image, and wherein the operations further comprise: controlling an autonomous vehicle based at least in part on detecting the object. ### Claim 21 (depends on 20) The one or more non-transitory computer-readable media of claim 20, wherein the difference is based at least in part on at least one of i) a first optical flow determined from the first data and the second data, ii) a second optical flow determined from a first feature map associated with the first data and a second feature map associated with the second data, or iii) tracking of a bounding box corresponding to the object and determined from the first data and the second data.