Scene selection method, scene selection apparatus and program

A scene selection method comprising:

• when S is a number of scenes, M is a number of input acoustic signals, and N is a number of input video signals,
• an acoustic signal encoding step of generating, by a scene selection device, an integrated acoustic feature amount from an m-th input acoustic signal (m=1, . . . , M) and a position where the m-th input acoustic signal is acquired (hereinafter referred to as an m-th input acoustic signal acquisition position) (m=1, . . . , M), wherein the generating the integrated acoustic feature amount further comprises:
• an m-th acoustic encoding step (m=1, . . . , M) of generating an m-th acoustic feature amount from the m-th input acoustic signal;
• an m-th conditional acoustic encoding step (m=1, . . . , M) of generating an m-th conditional acoustic feature amount from the m-th acoustic feature amount and the m-th input acoustic signal acquisition position, a dimension of the m-th conditional acoustic feature amount being smaller than a dimension of the m-th acoustic feature amount, thereby removing a feature that is redundant according to a position of acquiring the m-th input acoustic signal;
• and
• an integrated acoustic encoding step of generating the integrated acoustic feature amount from them-th conditional acoustic feature amount (m=1, . . . , M);
• a video signal encoding step of generating, by the scene selection device, an integrated video feature amount from an n-th input video signal (n=1, . . . , N) and a position where the n-th input video signal is acquired (hereinafter referred to as an n-th input video signal acquisition position) (n=1, . . . , N)
• wherein the generating the integrated video feature amount further comprises:
• an n-th video encoding step (n=1, . . . , N) of generating an n-th video feature amount from then-th input video signal;
• an n-th conditional video encoding step (n=1, . . . , N) of generating an n-th conditional video feature amount from the n-th video feature amount and the n-th input video signal acquisition position, a dimension of the n-th conditional video feature amount being smaller than a dimension of the n-th video feature amount, thereby removing a feature that is redundant according to a position of acquiring the n-th input video signal;
• and
• an integrated video encoding step of generating the integrated video feature amount from the n-th conditional video feature amount (n=1, . . . , N);
• and
• a scene selection step of selecting, by the scene selection device, a scene from which M input acoustic signals and N input video signals are acquired from among S scenes, using the integrated acoustic feature amount and the integrated video feature amount, wherein
• the m-th acoustic encoding step (m=1, . . . , M) is performed by a multi-layer Convolutional Neural Network (CNN) and the input to the multi-layer CNN is a logarithmic mel spectrogram obtained by applying a short-time Fourier transform (STFT) to the m-th input acoustic signal and by further applying a mel filter bank,
• the m-th conditional acoustic encoding step (m=1, . . . , M) is performed by a first neural network comprising a linear layer and the input to the first neural network is a first vector obtained by combining the m-th acoustic feature amount and the m-th input acoustic signal acquisition position,
• the integrated acoustic encoding step is performed by a second neural network comprising a linear layer and the input to the second neural network is a second vector obtained by combining the m-th conditional acoustic feature amount (m=1, . . . , M),
• the n-th video encoding step (n=1, . . . , N) is performed by ResNet,
• the n-th conditional video encoding step (n=1, . . . , N) is performed by a third neural network comprising a linear layer and the input to the third neural network is a third vector obtained by combining the n-th video feature amount and the n-th input video signal acquisition position,
• the integrated video encoding step is performed by a fourth neural network comprising a linear layer and the input to the fourth neural network is a fourth vector obtained by combining the n-th conditional video feature amount (n=1, . . . , N), and
• the scene selection step is performed by a fifth neural network comprising a linear layer and a Softmax layer and the input to the fifth neural network is a fifth vector obtained by combining the integrated acoustic feature amount and the integrated video feature amount.

A scene selection device comprising:

• when S is a number of scenes and M is a number of input acoustic signals,
• an acoustic signal encoding circuitry configured to generate an integrated acoustic feature amount from an m-th input acoustic signal (m=1, . . . , M) and a position where the m-th input acoustic signal is acquired (hereinafter referred to as an m-th input acoustic signal acquisition position) (m=1, . . . , M), wherein the acoustic signal encoding circuitry further comprises:
• an m-th acoustic encoding circuitry (m=1, . . . , M) configured to generate an m-th acoustic feature amount from the m-th input acoustic signal,
• an m-th conditional acoustic encoding circuitry (m=1, . . . , M) configured to generate an m-th conditional acoustic feature amount from the m-th acoustic feature amount and the m-th input acoustic signal acquisition position, a dimension of the m-th conditional acoustic feature amount being smaller than a dimension of the m-th acoustic feature amount, thereby removing a feature that is redundant according to a position of acquiring the m-th input acoustic signal, and
• an integrated acoustic encoding circuitry configured to generate the integrated acoustic feature amount from the m-th conditional acoustic feature amount (m=1, . . . , M);
• and
• a scene selection circuitry configured to select a scene from which M input acoustic signals are acquired from among S scenes, using the integrated acoustic feature amount, wherein
• the m-th acoustic encoding circuitry (m=1, . . . , M) is configured using a multi-layer Convolutional Neural Network (CNN) and the input to the multi-layer CNN is a logarithmic mel spectrogram obtained by applying a short-time Fourier transform (STFT) to the m-th input acoustic signal and by further applying a mel filter bank,
• the m-th conditional acoustic encoding circuitry (m=1, . . . , M) is configured using a first neural network comprising a linear layer and the input to the first neural network is a first vector obtained by combining the m-th acoustic feature amount and the m-th input acoustic signal acquisition position,
• the integrated acoustic encoding circuitry is configured using a second neural network comprising a linear layer and the input to the second neural network is a second vector obtained by combining the m-th conditional acoustic feature amount (m=1, . . . , M), and
• the scene selection circuitry is configured using a third neural network comprising a linear layer and a Softmax layer.