Low-latency captioning system

A computer-executable training method for training a multimodal sensor feature encoder, a timing detector, and a scene caption decoder, comprising:

• providing a training dataset that includes a set of multimodal sensor signals and a set of groundtruth scene captions;
• converting the multimodal sensor signals to a feature vector sequence using a feature extractor;
• eliminating future frames from the feature vector sequence using a future frame eliminator, wherein the future frame eliminator takes first feature vectors from the feature vector sequence and removes other feature vectors to generates a future-eliminated feature vector sequence;
• encoding the future-eliminated feature vector sequence to a hidden activation vector sequence;
• training a low-latency scene caption generator by computing a loss value
• computing a loss value based on a posterior probability distribution and the groundtruth scene captions;
• training the timing detector by computing a supervision signal based on a scene caption similarity between a scene caption for the future-eliminated feature vector sequence and the groundtruth scene captions to detect whether the current timing is appropriate or not.

A video captioning system comprising:

• an interface configured to acquire a stream of audio-visual signals including image and sound data;
• a memory to store a computer-executable video captioning model including an audio-visual feature extractor, an audio-visual feature encoder, a timing detector, and a video caption decoder, wherein the audio-visual encoder is shared by the timing decoder and the timing detector and the video caption decoder;
• a processor, in connection with the memory, configured to perform steps of:
• extracting audio and visual features from the audio-visual signals by use of the audio-visual extractor;
• encoding the audio and visual features from the audio-visual signals by use of the audio-visual encoder;
• determining a timing of generating a video caption by use of the timing detector, wherein the timing detector is trained to detect whether the current timing is appropriate or not, wherein the timing is arranged an early stage of the stream of the audio-visual signals;
• and
• generating the video caption describing audio-visual scenes based on the audio and visual features by using the video caption decoder according to the timing.