Back to results
US 12,182,519 B2Granted
Systems and methods for semantic-based pre-training for dialogue understanding
AssigneeTencent America LLC
InventorLinfeng Song
Life of this patent
Estimating the term…
Abstract
Systems and methods for pre-training a dialogue model with semantic information include: generating a dialogue-level abstract meaning representation (AMR) graph for an input dialogue associated with a speaker, learning core semantic units of the input dialogue based on nodes of the dialogue-level AMR graph, learning semantic relations between words of a sentence of the input dialogue based on edges of the dialogue-level AMR graph, learning an overall agreement of the input dialogue and the dialogue-level AMR graph, and training the dialogue model based on the learned core semantic units, semantic relations between words, and overall agreement.
Claims15
A method for pre-training a dialogue model with semantic information performed by at least one processor, the method comprising:
- generating a dialogue-level abstract meaning representation (AMR) graph for an input dialogue associated with a speaker;
- learning core semantic units of the input dialogue as a first optimization task based on nodes of the dialogue-level AMR graph, wherein one or more tokens associated with the core semantic units are given more attention that one or more tokens not associated with the one or more core semantic units;
- learning semantic relations between words of a sentence of the input dialogue as a second optimization task based on edges of the dialogue-level AMR graph by projecting the edges of the dialogue-level AMR graph onto a corresponding sentence of the input dialogue according to a node-to-word alignment;
- learning an overall agreement of the input dialogue and the dialogue-level AMR graph as a third optimization task by linearizing the dialogue-level AMR graph, using a pre-trained encoder to transform the linearized AMR into a set of hidden states, and maximizing a similarity score between the set of hidden states and the dialogue-level AMR graph;
- and
- generating a trained dialogue model based on optimizing a total loss based on the first optimization task, the second optimization task, and the third optimization task and according to equation
- Ltotal=Lsem_mim+αLrel+βLsim,
- wherein α is a weighting hyper-parameters for Lrel, and β is a weighting hyper-parameters parameters for Lsim,
- wherein Lsim is a first training objective for learning the core semantic units, Lrel is a second training objective for learning the semantic relations, and wherein Lsim is a third training objective for learning the overall similarity.
6 more elements
An apparatus for pre-training a dialogue model with semantic information, the apparatus comprising:
- a memory storing computer programming code;
- and
- at least one processor configured to operate as instructed by the computer programming code, the computer programming code including:
- generating code configured to cause the at least one processor to generate a dialogue-level abstract meaning representation (AMR) graph for an input dialogue associated with a speaker;
- first learning code configured to cause the at least one processor to determine core semantic units of the input dialogue based on nodes of the dialogue-level AMR graph, wherein one or more tokens associated with the core semantic units are given more attention that one or more tokens not associated with the one or more core semantic units;
- second learning code configured to cause the at least one processor to determine semantic relations between words of a sentence of the input dialogue based on edges of the dialogue-level AMR graph by projecting the edges of the dialogue-level AMR graph onto a corresponding sentence of the input dialogue according to a node-to-word alignment;
- third learning code configured to cause the at least one processor to determine an overall agreement of the input dialogue and the dialogue-level AMR graph by linearizing the dialogue-level AMR graph, using a pre trained encoder to transform the linearized AMR into a set of hidden states, and maximizing a similarity score between the set of hidden states and the dialogue-level AMR graph;
- and
- training code configured to cause the at least one processor to generating a trained dialogue model based on optimizing a total loss based on the first optimization task, the second optimization task, and the third optimization task and according to equation
- Ltotal=Lsem_mim+αLrel+βLsim,
- wherein α is a weighting hyper-parameters for Lrel, and β is a weighting hyper-parameters for Lsim,
- wherein Lsim is a first training objective for learning the core semantic units, Lrel is a second training objective for learning the semantic relations, and wherein Lsim is a third training objective for learning the overall similarity.
9 more elements
A non-transitory computer-readable recording medium having recorded thereon a computer program which, when executed by a processor, causes the processor to:
- generate a dialogue-level abstract meaning representation (AMR) graph for an input dialogue associated with a speaker;
- determine core semantic units of the input dialogue based on nodes of the dialogue-level AMR graph, wherein one or more tokens associated with the core semantic units are given more attention that one or more tokens not associated with the one or more core semantic units;
- determine semantic relations between words of a sentence of the input dialogue based on edges of the dialogue-level AMR graph by projecting the edges of the dialogue-level AMR graph onto a corresponding sentence of the input dialogue according to a node-to-word alignment;
- determine an overall agreement of the input dialogue and the dialogue-level AMR graph by linearizing the dialogue-level AMR graph, using a pre-trained encoder to transform the linearized AMR into a set of hidden states, and maximizing a similarity score between the set of hidden states and the dialogue-level AMR graph;
- and
- generate a trained dialogue model based on the optimizing a total loss based on the first optimization task, the second optimization task, and the third optimization task and according to equation
- Ltotal=Lsem_mim+αLrel+βLsim,
- wherein α is a weighting hyper-parameters for Lrel, and β is a weighting hyper-parameters for Lsim,
- wherein Lsim is a first training objective for learning the core semantic units, Lrel is a second training objective for learning the semantic relations, and wherein Lsim is a third training objective for learning the overall similarity.
6 more elements
Dependencies are inferred from the claim text.