Deliberation model-based two-pass end-to-end speech recognition

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a first-pass hypothesis and an encoded acoustic frame, the first-pass hypothesis generated by a recurrent neural network (RNN) decoder model for the encoded acoustic frame;
• encoding the first-pass hypothesis at a hypothesis encoder;
• generating, using an acoustic attention mechanism attending to the encoded acoustic frame, a first context vector;
• generating, using a hypothesis attention mechanism attending to the encoded first-pass hypothesis, a second context vector;
• and
• decoding the first context vector and the second context vector at a context vector decoder to form a second-pass hypothesis.