End-to-end speech recognition adapted for multi-speaker applications

An end-to-end automatic speech recognition (ASR) system, comprising:

• a processor;
• and a memory having instructions stored thereon, wherein the processor is configured to execute the stored instructions to cause the ASR system to:
• collect a sequence of acoustic frames providing a digital representation of an acoustic signal including a mixture of speeches performed by multiple speakers;
• encode each frame from the sequence of acoustic frames with a multi-head encoder encoding each frame into a likelihood of a transcription output and a likelihood of an identity of a speaker to produce a sequence of likelihoods of transcription outputs and a sequence of likelihoods of identities of the speakers corresponding to the sequence of acoustic frames;
• decode the sequence of likelihoods of transcription outputs and the sequence of likelihoods of identities of the speakers with a decoder performing an alignment producing a sequence of transcription outputs annotated with identities of the speakers, wherein the multi-head encoder and the decoder form at least a part of a neural network trained with an extended connectionist temporal classification (CTC) objective function to enforce an alignment between an input of the neural network and an output of the neural network on a graph with nodes indicative of transcription outputs and edges indicative of speaker transitions;
• and
• submit the sequence of transcription outputs annotated with the identities of the speakers into a downstream application.