Model compression using cycle generative adversarial network knowledge distillation

A computer-implemented method, comprising:

• at an electronic device with one or more processors and memory:
• receiving, at a first portion of a language model, an input;
• obtaining, based on the input, a first output distribution from the first portion of the language model;
• adjusting, using a first training model, the language model based on the first output distribution;
• receiving, at a second portion of the language model, the first output distribution;
• obtaining, based on the first output distribution, a first representation of the input from the second portion of the language model;
• adjusting, using a second training model, the language model based on the first representation of the input;
• and
• providing, using the adjusted language model, an output based on a received user input.

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

• receive, at a first portion of a language model, an input;
• obtain, based on the input, a first output distribution from the first portion of the language model;
• adjust, using a first training model, the language model based on the first output distribution;
• receive, at a second portion of the language model, the first output distribution;
• obtain, based on the first output distribution, a first representation of the input from the second portion of the language model;
• adjust, using a second training model, the language model based on the first representation of the input;
• and
• provide, using the adjusted language model, an output based on a received user input.