End-To-End Multi-Talker Overlapping Speech Recognition

A system for training a speech recognition model with a loss function, the system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a training example comprising an audio signal including a first segment corresponding to audio spoken by a first speaker, a second segment corresponding to audio spoken by a second speaker, and an overlapping region where the first segment overlaps the second segment, the overlapping region comprising a known start time and a known end time;
• for each of the first speaker and the second speaker, generating a respective masked audio embedding based on the training sample;
• determining whether the first speaker was speaking:
• prior to the known start time of the overlapping region;
• or
• after the known end time of the overlapping region;
• when the first speaker was speaking prior to the known start time of the overlapping region, applying, to the respective masked audio embedding for the first speaker, a first masking loss after the known end time;
• and
• when the first speaker was speaking after the known end time of the overlapping region, applying, to the respective masked audio embedding for the first speaker, the first masking loss before the known start time.