Spatiotemporal recycling network

An apparatus for processing one or more frames, comprising:

• at least one memory;
• and
• one or more processors communicatively coupled to the at least one memory, the one or more processors being configured to:
• obtain video data comprising a current frame and one or more reference frames;
• determine, based on a comparison of the current frame and the one or more reference frames, a difference between the current frame and the one or more reference frames;
• based on the difference being below a threshold, determine to perform semantic segmentation of the current frame using a first neural network;
• based on determining to perform the semantic segmentation of the current frame using the first neural network, obtain lower-spatial resolution features from storage, wherein the lower-spatial resolution features are extracted from multiple reference frames by a second neural network, the first neural network having at least one of a smaller structure and a lower processing cost than the second neural network;
• pool the lower-spatial resolution features to generate pooled lower-spatial resolution features, the pooled lower-spatial resolution features representing the lower-spatial resolution features extracted from the multiple reference frames;
• determine a set of features associated with the current frame by combining the pooled lower-spatial resolution features with higher-spatial resolution features extracted from the current frame by the first neural network;
• generate a segmentation output using the first neural network based on the set of features associated with the current frame;
• and
• update the one or more reference frames with the current frame and perform semantic segmentation determination of future frames using the first neural network or the second neural network based on the updated one or more reference frames.

At least one non-transitory computer-readable storage medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to:

• obtain video data comprising a current frame and one or more reference frames;
• determine, based on a comparison of the current frame and the one or more reference frames, a difference between the current frame and the one or more reference frames;
• based on the difference being below a threshold, determine to perform, using a first neural network, semantic segmentation of the current frame;
• based on determining to perform the semantic segmentation of the current frame using the first neural network, obtain lower-spatial resolution features from storage, wherein the lower-spatial resolution features are extracted from multiple reference frames by a second neural network, the first neural network having at least one of a smaller structure and a lower processing cost than the second neural network;
• pool the lower-spatial resolution features to generate pooled lower-spatial resolution features, the pooled lower-spatial resolution features representing the lower-spatial resolution features extracted from the multiple reference frames;
• determine a set of features associated with the current frame by combining the pooled lower-spatial resolution features with higher-spatial resolution features extracted from the current frame by the first neural network;
• generate a segmentation output using the first neural network based on the set of features associated with the current frame;
• and
• update the one or more reference frames with the current frame and perform semantic segmentation determination of future frames using the first neural network or the second neural network based on the updated one or more reference frames.

A method of processing one or more frames, comprising:

• obtaining video data including a current frame and one or more reference frames;
• obtaining sensor data from one or more sensors;
• determining, based on a comparison of the current frame, the one or more reference frames, and the sensor data, a difference between the current frame and the one or more reference frames
• based on the difference being below a threshold, determining to perform semantic segmentation of the current frame using a first neural network;
• based on determining to perform the semantic segmentation of the current frame using the first neural network, obtaining lower-spatial resolution features from storage, wherein the lower-spatial resolution features are extracted from multiple reference frames by a second neural network, the first neural network having at least one of a smaller structure and a lower processing cost than the second neural network;
• pooling the lower-spatial resolution features to generate pooled lower-spatial resolution features, the pooled lower-spatial resolution features representing the lower-spatial resolution features extracted from the multiple reference frames;
• determining a set of features associated with the current frame by combining the pooled lower-spatial resolution features with higher-spatial resolution features extracted from the current frame by the first neural network;
• and
• generating a segmentation output using the first neural network based on the set of features associated with the current frame.