Multi-modal framework for multi-channel target speech separation
AssigneeTencent America LLC
InventorsShi-Xiong Zhang, Yong Xu, Meng Yu, Dong Yu
Life of this patent
Estimating the term…
Abstract
A method, computer program, and computer system for separating a target voice from among a plurality of speakers is provided. Video data associated with the plurality of speakers and audio data associated with each of the one or more speakers are received. Video feature data is extracted from the received video data. The target voice is identified from among the plurality of speakers based on the received audio data and the extracted video feature data.
Claims20
A method of separating a target voice from among a plurality of speakers, the method comprising:
- receiving video data associated with the plurality of speakers;
- receiving audio data associated with each of the one or more speakers, the audio data comprising first audio data, synchronized to the video data, and second audio data comprising an enrollment utterance of at least one of the speakers pre-recorded prior to recording the video data;
- extracting video feature data from the received video data;
- identifying the target voice from among the plurality of speakers based on a mask generated by using inputs comprising at least the first audio data, the second audio data, and the extracted video feature data.
A computer system for separating a target voice from among a plurality of speakers, the computer system comprising:
- one or more computer-readable non-transitory storage media configured to store computer program code;
- and
- one or more computer processors configured to access said computer program code and operate as instructed by said computer program code, said computer program code including:
- first receiving code configured to cause the one or more computer processors to receive video data associated with the plurality of speakers;
- second receiving code configured to cause the one or more computer processors to receive audio data associated with each of the one or more speakers, the audio data comprising first audio data, synchronized to the video data, and second audio data comprising an enrollment utterance of at least one of the speakers pre-recorded prior to recording the video data;
- extracting code configured to cause the one or more computer processors to extract video feature data from the received video data;
- identifying code configured to cause the one or more computer processors to identify the target voice from among the plurality of speakers based on a mask generated by using inputs comprising at least the first audio data, the second audio data, and the extracted video feature data.
A non-transitory computer readable medium having stored thereon a computer program for separating a target voice from among a plurality of speakers, the computer program configured to cause one or more computer processors to:
- receive video data associated with the plurality of speakers;
- receive audio data associated with each of the one or more speakers, the audio data comprising first audio data, synchronized to the video data, and second audio data comprising an enrollment utterance of at least one of the speakers pre-recorded prior to recording the video data;
- extract video feature data from the received video data;
- identify the target voice from among the plurality of speakers based on a mask generated by using inputs comprising at least the first audio data, the second audio data, and the extracted video feature data.
Dependencies are inferred from the claim text.