# US 2019/0244367 A1 · Systems and methods for determining optical flow > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0244367 A1 - Document ID: US20190244367A1 - Status: Published application - Kind: A1 - Application number: 16/211,104 - Filed: 2018-12-05 - Published: 2019-08-08 - Earliest claimed: 2015-12-28 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Balamanohar Paluri - Web page: https://patentgrep.com/patent/US20190244367A1 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling} - G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06T 2207/10016: Video; Image sequence - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems, methods, and non-transitory computer-readable media can obtain a first video frame and a second video frame. The first video frame can be processed using a convolutional neural network to output a first set of feature maps. The second video frame can be processed using the convolutional neural network to output a second set of feature maps. The first set of feature maps and the second set of feature maps can be processed using a spatial matching layer of the convolutional neural network to determine an optical flow for at least one pixel. ## Claims ### Claim at position 1 1-20. (canceled) ### Claim 21 A computer-implemented method comprising: obtaining, by a computing system, a first video frame and a second video frame; processing, by the computing system, the first video frame using a convolutional neural network to output a first set of feature maps, wherein the convolutional neural network includes at least a first convolutional layer and a second convolutional layer, wherein signals outputted by the first convolutional layer are processed by a first pooling layer and then inputted to the second convolutional layer; processing, by the computing system, the second video frame using the convolutional neural network to output a second set of feature maps; processing, by the computing system, the first set of feature maps and the second set of feature maps by the convolutional neural network to determine an optical flow for at least one pixel. ### Claim 22 (depends on 21) The computer-implemented method of claim 21, further comprising: categorizing, by the computing system, at least one physical action represented in the first video frame and in the second video frame based at least in part on the optical flow. ### Claim 23 (depends on 21) The computer-implemented method of claim 21, further comprising: categorizing, by the computing system, at least one sport being played as represented in the first video frame and in the second video frame based at least in part on the optical flow. ### Claim 24 (depends on 21) The computer-implemented method of claim 21, further comprising: determining, by the computing system, information describing at least one object represented in the first video frame and in the second video frame based at least in part on the optical flow, the information including a corresponding direction and magnitude for the at least one object. ### Claim 25 (depends on 21) The computer-implemented method of claim 21, wherein the first set of feature maps and the second set of feature maps are processed using a spatial matching layer of the convolutional neural network. ### Claim 26 (depends on 25) The computer-implemented method of claim 25, wherein the spatial matching layer performs a pairwise analysis of the first set of feature maps and the second set of feature maps. ### Claim 27 (depends on 21) The computer-implemented method of claim 21, further comprising: determining, by the computing system, at least one inaccuracy in the optical flow for the pixel based on a ground truth optical flow for the pixel; and adjusting, by the computing system, one or more weight values of one or more filters associated with the convolutional neural network to minimize the inaccuracy. ### Claim 28 (depends on 21) The computer-implemented method of claim 21, further comprising: processing, by the computing system, the first set of feature maps and the second set of feature maps using a different convolutional neural network that has been trained to determine optical flow for objects to determine an optical flow for at least one object. ### Claim 29 (depends on 21) The computer-implemented method of claim 21, wherein the optical flow is determined based at least in part on an evaluation of displacement measurements corresponding to the pixel across one or more feature maps in the first set of feature maps and one or more feature maps in the second set of feature maps. ### Claim 30 (depends on 21) The computer-implemented method of claim 21, wherein the optical flow for the pixel provides at least a predicted direction or magnitude of the pixel along an x-axis and a predicted direction or magnitude of the pixel along a y-axis. ### Claim 31 A system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform: obtaining a first video frame and a second video frame; processing the first video frame using a convolutional neural network to output a first set of feature maps, wherein the convolutional neural network includes at least a first convolutional layer and a second convolutional layer, wherein signals outputted by the first convolutional layer are processed by a first pooling layer and then inputted to the second convolutional layer; processing the second video frame using the convolutional neural network to output a second set of feature maps; processing the first set of feature maps and the second set of feature maps by the convolutional neural network to determine an optical flow for at least one pixel. ### Claim 32 (depends on 31) The system of claim 31, wherein the instructions further cause the system to perform: categorizing at least one physical action represented in the first video frame and in the second video frame based at least in part on the optical flow. ### Claim 33 (depends on 31) The system of claim 31, wherein the instructions further cause the system to perform: categorizing at least one sport being played as represented in the first video frame and in the second video frame based at least in part on the optical flow. ### Claim 34 (depends on 31) The system of claim 31, wherein the instructions further cause the system to perform: determining information describing at least one object represented in the first video frame and in the second video frame based at least in part on the optical flow, the information including a corresponding direction and magnitude for the at least one object. ### Claim 35 (depends on 31) The system of claim 31, wherein the first set of feature maps and the second set of feature maps are processed using a spatial matching layer of the convolutional neural network. ### Claim 36 A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising: obtaining a first video frame and a second video frame; processing the first video frame using a convolutional neural network to output a first set of feature maps, wherein the convolutional neural network includes at least a first convolutional layer and a second convolutional layer, wherein signals outputted by the first convolutional layer are processed by a first pooling layer and then inputted to the second convolutional layer; processing the second video frame using the convolutional neural network to output a second set of feature maps; processing the first set of feature maps and the second set of feature maps by the convolutional neural network to determine an optical flow for at least one pixel. ### Claim 37 (depends on 36) The non-transitory computer-readable storage medium of claim 36, wherein the instructions further cause the computing system to perform: categorizing at least one physical action represented in the first video frame and in the second video frame based at least in part on the optical flow. ### Claim 38 (depends on 36) The non-transitory computer-readable storage medium of claim 36, wherein the instructions further cause the computing system to perform: categorizing at least one sport being played as represented in the first video frame and in the second video frame based at least in part on the optical flow. ### Claim 39 (depends on 36) The non-transitory computer-readable storage medium of claim 36, wherein the instructions further cause the computing system to perform: determining information describing at least one object represented in the first video frame and in the second video frame based at least in part on the optical flow, the information including a corresponding direction and magnitude for the at least one object. ### Claim 40 (depends on 36) The non-transitory computer-readable storage medium of claim 36, wherein the first set of feature maps and the second set of feature maps are processed using a spatial matching layer of the convolutional neural network.