US 2021/0174093 A1Published application

Video action segmentation by mixed temporal domain adaption

AssigneeBaidu USA LLC

InventorsBaopu LI, Min-Hung CHEN, Yingze BAO

Life of this patent

Estimating the term…

Loading patent…

Abstract

Embodiments herein treat the action segmentation as a domain adaption (DA) problem and reduce the domain discrepancy by performing unsupervised DA with auxiliary unlabeled videos. In one or more embodiments, to reduce domain discrepancy for both the spatial and temporal directions, embodiments of a Mixed Temporal Domain Adaptation (MTDA) approach are presented to jointly align frame-level and video-level embedded feature spaces across domains, and, in one or more embodiments, further integrate with a domain attention mechanism to focus on aligning the frame-level features with higher domain discrepancy, leading to more effective domain adaptation. Comprehensive experiment results validate that embodiments outperform previous state-of-the-art methods. Embodiments can adapt models effectively by using auxiliary unlabeled videos, leading to further applications of large-scale problems, such as video surveillance and human activity analysis.

Claims20

Dependent claims

A computer-implemented method for training a video segmentation system for assigning a set of action labels to frames of a video, the method comprising:

inputting, for each input video from a first set of video data and from a second set of video data, a set of frame-level features of frames of the input video into a video segmentation network, the first set of video data comprising videos with associated action labels, the second set of video data comprising videos without associated action labels, and the video segmentation network comprising:
at least one domain adaption temporal convolution network, which comprises:
a multi-layer temporal convolution network that receives an input related to the set of frame-level features of the input video and outputs a set of spatio-temporal-refined frame-level features;
a classification layer that receives the set of spatio-temporal-refined frame-level features and outputs a set of frame-level predictions;
a local temporal domain adaptation model that receives the set of spatio-temporal-refined frame-level features and outputs a domain prediction whether the set of spatio-temporal-refined frame-level features are from an input video from the first set of video data or the second set of video data;
a domain attentive pooling component that receives the set of spatio-temporal-refined frame-level features and the domain prediction and uses domain attention weights to combine the set of spatio-temporal-refined frame-level features into a video-level feature;
and
a global temporal domain adaptation model that receives the video-level feature and outputs a video-level domain prediction whether the video-level feature is from an input video from the first set of video data or the second set of video data;
outputting, for each input video from the first set of video data and from the second set of video data, a final set of frame-level predictions in which each frame of at least some of the frames from a set of frames of the input video has an associated label prediction;
computing losses for the video segmentation network, the computed losses comprising at least one of the following:
responsive to an input video being from the first set of video data, a prediction loss related to the final set of frame-level predictions relative to the associated action labels for the input video;
a local domain loss, which represents error in predicting whether the set of spatio-temporal-refined frame-level features are from an input video from the first set of video data or the second set of video data;
and
a global domain loss, which represents error in predicting whether the video-level feature is from an input video from the first set of video data or the second set of video data;
and
updating the video segmentation network using the computed losses.

13 more elements

A computer-implemented method for using a video segmentation system to assign labels to at least some of the frames of a video, the method comprising:

inputting the video into the video segmentation system comprising:
a feature generator that receives the video and converts the video into a set of frame-level features;
a video segmentation network comprising at least one domain adaption temporal convolution network, which comprises:
a multi-layer temporal convolution network that receives an input related to the set of frame-level features of the input video and outputs a set of spatio-temporal-refined frame-level features;
a classification layer that receives the set of spatio-temporal-refined frame-level features and outputs a set of frame-level predictions;
and
outputting, for the video, a final set of frame-level predictions in which each frame of at least some of the frames from the set of frames of the video has an associated label prediction,
wherein the at least one domain adaption temporal convolution network was trained by performing steps comprising:
inputting, for each input video from a first set of video data and from a second set of video data, a set of frame-level features of frames of the input video into the video segmentation network, the first set of video data comprising videos with associated action labels, the second set of video data comprising videos without associated action labels, and the video segmentation network comprising:
at least one domain adaption temporal convolution network, which comprises:
a multi-layer temporal convolution network that receives an input related to the set of frame-level features of the input video and outputs a set of spatio-temporal-refined frame-level features;
a classification layer that receives the set of spatio-temporal-refined frame-level features and outputs a set of frame-level predictions;
a local temporal domain adaptation model that receives the set of spatio-temporal-refined frame-level features and outputs a domain prediction whether the set of spatio-temporal-refined frame-level features are from an input video from the first set of video data or the second set of video data;
a domain attentive pooling component that receives the set of spatio-temporal-refined frame-level features and the domain prediction and uses domain attention weights to combine the set of spatio-temporal-refined frame-level features into a video-level feature;
and
a global temporal domain adaptation model that receives the video-level feature and outputs a video-level domain prediction whether the video-level feature is from an input video from the first set of video data or the second set of video data;
outputting, for each input video from the first set of video data and from the second set of video data, a final set of frame-level predictions in which each frame of at least some of the frames from a set of frames of the input video has an associated label prediction;
computing losses for the video segmentation network, the computed losses comprising at least one of the following:
responsive to an input video being from the first set of video data, a prediction loss related to the final set of frame-level predictions relative to the associated action labels for the input video;
a local domain loss, which represents error in predicting whether the set of spatio-temporal-refined frame-level features are from an input video from the first set of video data or the second set of video data;
and
a global domain loss, which represents error in predicting whether the video-level feature is from an input video from the first set of video data or the second set of video data;
and
updating the video segmentation network using the computed losses.

21 more elements

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes steps for training a video segmentation system for assigning a set of action labels to frames of a video, the method comprising:

inputting, for each input video from a first set of video data and from a second set of video data, a set of frame-level features of frames of the input video into a video segmentation network, the first set of video data comprising videos with associated action labels, the second set of video data comprising videos without associated action labels, and the video segmentation network comprising:
at least one domain adaption temporal convolution network, which comprises:
a multi-layer temporal convolution network that receives an input related to the set of frame-level features of the input video and outputs a set of spatio-temporal-refined frame-level features;
a classification layer that receives the set of spatio-temporal-refined frame-level features and outputs a set of frame-level predictions;
a local temporal domain adaptation model that receives the set of spatio-temporal-refined frame-level features and outputs a domain prediction whether the set of spatio-temporal-refined frame-level features are from an input video from the first set of video data or the second set of video data;
a domain attentive pooling component that receives the set of spatio-temporal-refined frame-level features and the domain prediction and uses domain attention weights to combine the set of spatio-temporal-refined frame-level features into a video-level feature;
and
a global temporal domain adaptation model that receives the video-level feature and outputs a video-level domain prediction whether the video-level feature is from an input video from the first set of video data or the second set of video data;
outputting, for each input video from the first set of video data and from the second set of video data, a final set of frame-level predictions in which each frame of at least some of the frames from a set of frames of the input video has an associated label prediction;
computing losses for the video segmentation network, the computed losses comprising at least one of the following:
responsive to an input video being from the first set of video data, a prediction loss related to the final set of frame-level predictions relative to the associated action labels for the input video;
a local domain loss, which represents error in predicting whether the set of spatio-temporal-refined frame-level features are from an input video from the first set of video data or the second set of video data;
and
a global domain loss, which represents error in predicting whether the video-level feature is from an input video from the first set of video data or the second set of video data;
and
updating the video segmentation network using the computed losses.

13 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/706,590Filed 6 Dec 2019
Published: 10 Jun 2021
Term ends: Not estimatedThe source records don’t support an estimate.

Classification19 codes

G06V 20/49
In video content: segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or…
G06F 18/217
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: validation; Performance…
G06N 3/045
Architecture, e.g. interconnection topology: combinations of networks
G06N 3/0464
Architecture, e.g. interconnection topology: convolutional networks [CNN, ConvNet]

Family

US 11,138,441 B2Related patent
US 2021/0174093 A1This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key