# US 12,182,519 B2 · Systems and methods for semantic-based pre-training for dialogue understanding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,519 B2 - Document ID: US12182519B2 - Status: Granted - Kind: B2 - Application number: 17/887,134 - Filed: 2022-08-12 - Published: 2024-02-15 - Granted: 2024-12-31 - Earliest claimed: 2022-08-12 - Assignees: Tencent America LLC - Applicants: Tencent America LLC - Inventors: Linfeng Song - Web page: https://patentgrep.com/patent/US12182519B2 ## CPC - G06F 40/35: Discourse or dialogue representation - G06F 40/205: Parsing - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems and methods for pre-training a dialogue model with semantic information include: generating a dialogue-level abstract meaning representation (AMR) graph for an input dialogue associated with a speaker, learning core semantic units of the input dialogue based on nodes of the dialogue-level AMR graph, learning semantic relations between words of a sentence of the input dialogue based on edges of the dialogue-level AMR graph, learning an overall agreement of the input dialogue and the dialogue-level AMR graph, and training the dialogue model based on the learned core semantic units, semantic relations between words, and overall agreement. ## Claims ### Claim 1 A method for pre-training a dialogue model with semantic information performed by at least one processor, the method comprising: generating a dialogue-level abstract meaning representation (AMR) graph for an input dialogue associated with a speaker; learning core semantic units of the input dialogue as a first optimization task based on nodes of the dialogue-level AMR graph, wherein one or more tokens associated with the core semantic units are given more attention that one or more tokens not associated with the one or more core semantic units; learning semantic relations between words of a sentence of the input dialogue as a second optimization task based on edges of the dialogue-level AMR graph by projecting the edges of the dialogue-level AMR graph onto a corresponding sentence of the input dialogue according to a node-to-word alignment; learning an overall agreement of the input dialogue and the dialogue-level AMR graph as a third optimization task by linearizing the dialogue-level AMR graph, using a pre-trained encoder to transform the linearized AMR into a set of hidden states, and maximizing a similarity score between the set of hidden states and the dialogue-level AMR graph; and generating a trained dialogue model based on optimizing a total loss based on the first optimization task, the second optimization task, and the third optimization task and according to equation Ltotal=Lsem_mim+αLrel+βLsim, wherein α is a weighting hyper-parameters for Lrel, and β is a weighting hyper-parameters parameters for Lsim, wherein Lsim is a first training objective for learning the core semantic units, Lrel is a second training objective for learning the semantic relations, and wherein Lsim is a third training objective for learning the overall similarity. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the dialogue-level AMR graph for the input dialogue associated with the speaker comprises: building utterance-level AMR graphs by independently transforming utterances in the input dialogue into AMR using a pre-trained AMR parser; and connecting the utterance-level AMR graphs with a root node, wherein the edges of the dialogue-level AMR graph are labeled with the speaker. ### Claim 3 (depends on 1) The method of claim 1, wherein learning the core semantic units of the input dialogue based on the nodes of the dialogue-level AMR graph comprises: identifying one or more semantic-aware units of the input dialogue based on the nodes of the dialogue-level AMR graph; and increasing an attention given by the dialogue model to tokens of the input dialogue that correspond to the one or more semantic-aware units. ### Claim 4 (depends on 3) The method of claim 3, wherein identifying the one or more semantic-aware units of the input dialogue based on the nodes of the dialogue-level AMR graph comprises: identifying a token of the input dialogue that is aligned with a node of the dialogue-level AMR graph as a semantic-aware unit of the input dialogue. ### Claim 5 (depends on 3) The method of claim 3, wherein increasing the attention given by the dialogue model to tokens of the input dialogue that correspond to the one or more semantic-aware units comprises: assigning a masking probability to each token of the input dialogue, wherein the masking probability assigned to a token corresponding to the one or more semantic-aware units is higher than the masking probability assigned to other tokens of the input dialogue. ### Claim 6 (depends on 1) The method of claim 1, wherein the projecting the edges comprises: generating contextualized word hidden states of the input dialogue by using a Transformer encoder; and predicting relations between words based on the hidden states by using a deep biaffine neural parser. ### Claim 7 (depends on 1) The method of claim 1, wherein maximizing the similarity score between the set of hidden states and the dialogue-level AMR graph comprises: using a cosine similarity as a distance scoring operation; and adopting a contrastive learning framework to train the dialogue model. ### Claim 8 An apparatus for pre-training a dialogue model with semantic information, the apparatus comprising: a memory storing computer programming code; and at least one processor configured to operate as instructed by the computer programming code, the computer programming code including: generating code configured to cause the at least one processor to generate a dialogue-level abstract meaning representation (AMR) graph for an input dialogue associated with a speaker; first learning code configured to cause the at least one processor to determine core semantic units of the input dialogue based on nodes of the dialogue-level AMR graph, wherein one or more tokens associated with the core semantic units are given more attention that one or more tokens not associated with the one or more core semantic units; second learning code configured to cause the at least one processor to determine semantic relations between words of a sentence of the input dialogue based on edges of the dialogue-level AMR graph by projecting the edges of the dialogue-level AMR graph onto a corresponding sentence of the input dialogue according to a node-to-word alignment; third learning code configured to cause the at least one processor to determine an overall agreement of the input dialogue and the dialogue-level AMR graph by linearizing the dialogue-level AMR graph, using a pre trained encoder to transform the linearized AMR into a set of hidden states, and maximizing a similarity score between the set of hidden states and the dialogue-level AMR graph; and training code configured to cause the at least one processor to generating a trained dialogue model based on optimizing a total loss based on the first optimization task, the second optimization task, and the third optimization task and according to equation Ltotal=Lsem_mim+αLrel+βLsim, wherein α is a weighting hyper-parameters for Lrel, and β is a weighting hyper-parameters for Lsim, wherein Lsim is a first training objective for learning the core semantic units, Lrel is a second training objective for learning the semantic relations, and wherein Lsim is a third training objective for learning the overall similarity. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the generating code comprises: build code configured to cause the at least one processor to build utterance-level AMR graphs by independently transforming utterances in the input dialogue into AMR using a pre-trained AMR parser; and connect code configured to cause the at least one processor to connect the utterance-level AMR graphs with a root node, wherein the edges of the dialogue-level AMR graph are labeled with the speaker. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the learn code comprises: identify code configured to cause the at least one processor to identify one or more semantic-aware units of the input dialogue based on the nodes of the dialogue-level AMR graph; and attention increasing code configured to cause the at least one processor to increase an attention given by the dialogue model to tokens of the input dialogue that correspond to the one or more semantic-aware units. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein the identify code is configured to cause the at least one processor to identify a token of the input dialogue that is aligned with a node of the dialogue-level AMR graph as a semantic-aware unit of the input dialogue. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein the attention increasing code is configured to cause the at least one processor to assign a masking probability to each token of the input dialogue, wherein the masking probability assigned to a token corresponding to the one or more semantic-aware units is higher than the masking probability assigned to other tokens of the dialogue. ### Claim 13 (depends on 8) The apparatus of claim 8, wherein the training code is configured to cause the at least one processor to: generate contextualized word hidden states of the input dialogue by using a Transformer encoder; and predict relations between words based on the hidden states by using a deep biaffine neural parser. ### Claim 14 (depends on 8) The apparatus of claim 8, wherein the third learning code is configured to cause the at least one processor to: use a cosine similarity as a distance scoring operation; and adopt a contrastive learning framework to train the dialogue model. ### Claim 15 A non-transitory computer-readable recording medium having recorded thereon a computer program which, when executed by a processor, causes the processor to: generate a dialogue-level abstract meaning representation (AMR) graph for an input dialogue associated with a speaker; determine core semantic units of the input dialogue based on nodes of the dialogue-level AMR graph, wherein one or more tokens associated with the core semantic units are given more attention that one or more tokens not associated with the one or more core semantic units; determine semantic relations between words of a sentence of the input dialogue based on edges of the dialogue-level AMR graph by projecting the edges of the dialogue-level AMR graph onto a corresponding sentence of the input dialogue according to a node-to-word alignment; determine an overall agreement of the input dialogue and the dialogue-level AMR graph by linearizing the dialogue-level AMR graph, using a pre-trained encoder to transform the linearized AMR into a set of hidden states, and maximizing a similarity score between the set of hidden states and the dialogue-level AMR graph; and generate a trained dialogue model based on the optimizing a total loss based on the first optimization task, the second optimization task, and the third optimization task and according to equation Ltotal=Lsem_mim+αLrel+βLsim, wherein α is a weighting hyper-parameters for Lrel, and β is a weighting hyper-parameters for Lsim, wherein Lsim is a first training objective for learning the core semantic units, Lrel is a second training objective for learning the semantic relations, and wherein Lsim is a third training objective for learning the overall similarity.