Apparatus and method for generating video descriptions based on artificial neural networks

A method for generating video descriptions that is performed in a computing device including one or more processors and a memory storing one or more programs executed by the one or more processors, the method comprising:

• extracting, at a feature generating module, a plurality of preset features from a video and generating one synthetic feature based on the plurality of extracted features;
• and
• receiving, at a description generating module, the synthetic feature and outputting a description text for the video,
• wherein the generating of the synthetic feature includes:
• receiving, at a first feature extractor, the video and extracting spectral features from the video;
• receiving, at a second feature extractor, the video and extracting spatial features from the video;
• receiving, at a third feature extractor, the video and extracting optical flow features from the video;
• receiving, at a fourth feature extractor, one or more sentences describing the video and extracting text features from the sentences;
• and
• generating, at a synthesizer, the synthetic feature based on the spectral features, the spatial features, the optical flow features, and the text features,
• wherein the first feature extractor includes a plurality of sequentially connected Fourier transform neural networks so that an output of a Fourier transform neural network is used as an input of a next Fourier transform neural network, and
• the extracting of the spectral features includes:
• sequentially performing, at a Fourier convolution layer, a Fourier transform and an inverse Fourier transform on frames of the video to extract a first sub-feature;
• performing, at a standard convolution layer, a convolution on the frames of the video to extract a second sub-feature;
• and
• connecting, at a connection layer, the first sub-feature and the second sub-feature to generate the spectral features.

A computer program stored in a non-transitory computer readable storage medium, comprising one or more instructions that, when executed by a computing device having one or more processors, cause the computing device to perform operations of:

• extracting, at a feature generating module, a plurality of preset features from a video and generating one synthetic feature based on the plurality of extracted features;
• and
• receiving, at a description generating module, the synthetic feature and outputting a description text for the video,
• the generating of the synthetic feature includes:
• receiving, at a first feature extractor, the video and extracting spectral features from the video;
• receiving, at a second feature extractor, the video and extracting spatial features from the video;
• receiving, at a third feature extractor, the video and extracting optical flow features from the video;
• receiving, at a fourth feature extractor, one or more sentences describing the video and extracting text features from the sentences;
• and
• generating, at a synthesizer, the synthetic feature based on the spectral features, the spatial features, the optical flow features, and the text features,
• wherein the first feature extractor includes a plurality of sequentially connected Fourier transform neural networks so that an output of a Fourier transform neural network is used as an input of a next Fourier transform neural network, and
• the extracting of the spectral features includes:
• sequentially performing, at a Fourier convolution layer, a Fourier transform and an inverse Fourier transform on frames of the video to extract a first sub-feature;
• performing, at a standard convolution layer, a convolution on the frames of the video to extract a second sub-feature;
• and
• connecting, at a connection layer, the first sub-feature and the second sub-feature to generate the spectral features.