# US 12,118,810 B2 · Spatiotemporal recycling network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,810 B2 - Document ID: US12118810B2 - Status: Granted - Kind: B2 - Application number: 17/408,779 - Filed: 2021-08-23 - Published: 2022-02-24 - Granted: 2024-10-15 - Earliest claimed: 2020-08-24 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Yizhe Zhang; Amirhossein Habibian; Fatih Murat Porikli - Web page: https://patentgrep.com/patent/US12118810B2 ## CPC - G06T 7/11: Region-based segmentation - G06V 30/274: {Syntactic or semantic context, e.g. balancing} - G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06T 1/0007: {Image acquisition} - G06T 7/136: involving thresholding - G06T 7/174: involving the use of two or more images - G06T 7/215: Motion-based segmentation - G06V 10/40: Extraction of image or video features - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods} - G06V 10/806: {of extracted features} - G06V 10/82: using neural networks - G06V 10/95: {structured as a network, e.g. client-server architectures} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06V 10/759: {Region-based matching} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems, methods, and non-transitory media are provided for providing spatiotemporal recycling networks (e.g., for video segmentation). For example, a method can include obtaining video data including a current frame and one or more reference frames. The method can include determining, based on a comparison of the current frame and the one or more reference frames, a difference between the current frame and the one or more reference frames. Based on the difference being below a threshold, the method can include performing semantic segmentation of the current frame using a first neural network. The semantic segmentation can be performed based on higher-spatial resolution features extracted from the current frame by the first neural network and lower-resolution features extracted from the one or more reference frames by a second neural network. The first neural network has a smaller structure and/or a lower processing cost than the second neural network. ## Claims ### Claim 1 A method of processing one or more frames, comprising: obtaining video data including a current frame and one or more reference frames; determining, based on a comparison of the current frame and the one or more reference frames, a difference between the current frame and the one or more reference frames; based on the difference being below a threshold, determining to perform semantic segmentation of the current frame using a first neural network; based on determining to perform the semantic segmentation of the current frame using the first neural network, obtaining lower-spatial resolution features from storage, wherein the lower-spatial resolution features are extracted from multiple reference frames by a second neural network, the first neural network having at least one of a smaller structure and a lower processing cost than the second neural network; pooling the lower-spatial resolution features to generate pooled lower-spatial resolution features, the pooled lower-spatial resolution features representing the lower-spatial resolution features extracted from the multiple reference frames; determining a set of features associated with the current frame by combining the pooled lower-spatial resolution features with higher-spatial resolution features extracted from the current frame by the first neural network; generating a segmentation output using the first neural network based on the set of features associated with the current frame; and updating the one or more reference frames with the current frame and performing semantic segmentation determination of future frames using the first neural network or the second neural network based on the updated one or more reference frames. ### Claim 2 (depends on 1) The method of claim 1, wherein the second neural network comprises a segmentation network and the first neural network comprises a subnetwork structure of the segmentation network. ### Claim 3 (depends on 1) The method of claim 1, wherein the difference comprises a difference between pixel values associated with the current frame and pixel values associated with the one or more reference frames. ### Claim 4 (depends on 1) The method of claim 1, wherein the difference comprises an amount of change in motion between the current frame and the one or more reference frames. ### Claim 5 (depends on 1) The method of claim 1, wherein the comparison comprises generating a score using classification network, the score indicating a magnitude of the difference between the current frame and the one or more reference frames. ### Claim 6 (depends on 1) The method of claim 1, wherein the difference is based on a distance between one or more boundaries in a first boundary map associated with the current frame and one or more boundaries in a second boundary map associated with the one or more reference frames. ### Claim 7 (depends on 1) The method of claim 1, wherein the comparison comprises at least one of an image-level comparison and a region-level comparison, and wherein the difference is based on at least one of the image-level comparison and the region-level comparison. ### Claim 8 (depends on 1) The method of claim 1, wherein the second neural network comprises a multi-level network structure, wherein the multi-level network structure includes one or more deeper levels than a network structure associated with the first neural network, and wherein the lower-spatial resolution features are extracted at the one or more deeper levels of the multi-level network structure. ### Claim 9 (depends on 1) The method of claim 1, wherein the semantic segmentation is further based on one or more segmentation maps generated based on one or more previous frames. ### Claim 10 (depends on 1) The method of claim 1, further comprising: generating, based on the semantic segmentation, a segmentation output comprising one or more segmentation maps associated with the current frame. ### Claim 11 (depends on 1) The method of claim 1, wherein the higher-spatial resolution features are temporally variant relative to the lower-spatial resolution features. ### Claim 12 (depends on 1) The method of claim 1, further comprising: generating a feature update based on a convolution-based fusion of the pooled lower-spatial resolution features and the higher-spatial resolution features; and storing the feature update in the storage. ### Claim 13 (depends on 1) The method of claim 1, further comprising: determining, based on a second comparison of an additional frame and one or more additional reference frames, an additional difference between the additional frame and the one or more additional reference frames; and based on the additional difference being above the threshold, performing, using the second neural network, semantic segmentation of the additional frame. ### Claim 14 An apparatus for processing one or more frames, comprising: at least one memory; and one or more processors communicatively coupled to the at least one memory, the one or more processors being configured to: obtain video data comprising a current frame and one or more reference frames; determine, based on a comparison of the current frame and the one or more reference frames, a difference between the current frame and the one or more reference frames; based on the difference being below a threshold, determine to perform semantic segmentation of the current frame using a first neural network; based on determining to perform the semantic segmentation of the current frame using the first neural network, obtain lower-spatial resolution features from storage, wherein the lower-spatial resolution features are extracted from multiple reference frames by a second neural network, the first neural network having at least one of a smaller structure and a lower processing cost than the second neural network; pool the lower-spatial resolution features to generate pooled lower-spatial resolution features, the pooled lower-spatial resolution features representing the lower-spatial resolution features extracted from the multiple reference frames; determine a set of features associated with the current frame by combining the pooled lower-spatial resolution features with higher-spatial resolution features extracted from the current frame by the first neural network; generate a segmentation output using the first neural network based on the set of features associated with the current frame; and update the one or more reference frames with the current frame and perform semantic segmentation determination of future frames using the first neural network or the second neural network based on the updated one or more reference frames. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the second neural network comprises a segmentation network and the first neural network comprises a subnetwork structure of the segmentation network. ### Claim 16 (depends on 14) The apparatus of claim 14, wherein the difference comprises a difference between pixel values associated with the current frame and pixel values associated with the one or more reference frames. ### Claim 17 (depends on 14) The apparatus of claim 14, wherein the difference comprises an amount of change in motion between the current frame and the one or more reference frames. ### Claim 18 (depends on 14) The apparatus of claim 14, wherein the comparison comprises generating a score using classification network, the score indicating a magnitude of the difference between the current frame and the one or more reference frames. ### Claim 19 (depends on 14) The apparatus of claim 14, wherein the difference is based on a distance between one or more boundaries in a first boundary map associated with the current frame and one or more boundaries in a second boundary map associated with the one or more reference frames. ### Claim 20 (depends on 14) The apparatus of claim 14, wherein the comparison comprises at least one of an image-level comparison and a region-level comparison, and wherein the difference is based on at least one of the image-level comparison and the region-level comparison. ### Claim 21 (depends on 14) The apparatus of claim 14, wherein the second neural network comprises a multi-level network structure, wherein the multi-level network structure includes one or more deeper levels than a network structure associated with the first neural network, and wherein the lower-spatial resolution features are extracted at the one or more deeper levels of the multi-level network structure. ### Claim 22 (depends on 14) The apparatus of claim 14, wherein the semantic segmentation is further based on one or more segmentation maps generated based on one or more previous frames. ### Claim 23 (depends on 14) The apparatus of claim 14, the one or more processors being configured to: generate, based on the semantic segmentation, a segmentation output comprising one or more segmentation maps associated with the current frame. ### Claim 24 (depends on 14) The apparatus of claim 14, the one or more processors being configured to: generate a feature update based on a convolution-based fusion of the pooled lower-spatial resolution features and the higher-spatial resolution features; and store the feature update in the storage. ### Claim 25 (depends on 14) The apparatus of claim 14, the one or more processors being configured to: determine, based on a second comparison of an additional frame and one or more additional reference frames, an additional difference between the additional frame and the one or more additional reference frames; and based on the additional difference being above the threshold, perform, using the second neural network, semantic segmentation of the additional frame. ### Claim 26 (depends on 14) The apparatus of claim 14, wherein the apparatus is a mobile device. ### Claim 27 (depends on 14) The apparatus of claim 14, further comprising at least one of a camera and a display device. ### Claim 28 At least one non-transitory computer-readable storage medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to: obtain video data comprising a current frame and one or more reference frames; determine, based on a comparison of the current frame and the one or more reference frames, a difference between the current frame and the one or more reference frames; based on the difference being below a threshold, determine to perform, using a first neural network, semantic segmentation of the current frame; based on determining to perform the semantic segmentation of the current frame using the first neural network, obtain lower-spatial resolution features from storage, wherein the lower-spatial resolution features are extracted from multiple reference frames by a second neural network, the first neural network having at least one of a smaller structure and a lower processing cost than the second neural network; pool the lower-spatial resolution features to generate pooled lower-spatial resolution features, the pooled lower-spatial resolution features representing the lower-spatial resolution features extracted from the multiple reference frames; determine a set of features associated with the current frame by combining the pooled lower-spatial resolution features with higher-spatial resolution features extracted from the current frame by the first neural network; generate a segmentation output using the first neural network based on the set of features associated with the current frame; and update the one or more reference frames with the current frame and perform semantic segmentation determination of future frames using the first neural network or the second neural network based on the updated one or more reference frames. ### Claim 29 (depends on 28) The at least one non-transitory computer-readable storage medium of claim 28, wherein the second neural network comprises a segmentation network and the first neural network comprises a subnetwork structure of the segmentation network. ### Claim 30 A method of processing one or more frames, comprising: obtaining video data including a current frame and one or more reference frames; obtaining sensor data from one or more sensors; determining, based on a comparison of the current frame, the one or more reference frames, and the sensor data, a difference between the current frame and the one or more reference frames based on the difference being below a threshold, determining to perform semantic segmentation of the current frame using a first neural network; based on determining to perform the semantic segmentation of the current frame using the first neural network, obtaining lower-spatial resolution features from storage, wherein the lower-spatial resolution features are extracted from multiple reference frames by a second neural network, the first neural network having at least one of a smaller structure and a lower processing cost than the second neural network; pooling the lower-spatial resolution features to generate pooled lower-spatial resolution features, the pooled lower-spatial resolution features representing the lower-spatial resolution features extracted from the multiple reference frames; determining a set of features associated with the current frame by combining the pooled lower-spatial resolution features with higher-spatial resolution features extracted from the current frame by the first neural network; and generating a segmentation output using the first neural network based on the set of features associated with the current frame. ### Claim 31 (depends on 30) The method of claim 30, further comprising: updating features of the one or more reference frames with features of the current frame and performing semantic segmentation determination of future frames based on the updated features of the one or more reference frames.