# US 12,002,455 B2 · Semantically-augmented context representation generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,455 B2 - Document ID: US12002455B2 - Status: Granted - Kind: B2 - Application number: 17/383,284 - Filed: 2021-07-22 - Published: 2023-02-02 - Granted: 2024-06-04 - Earliest claimed: 2021-07-22 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Arvind Krishna Sridhar; Ravi Choudhary; Lae-Hoon Kim; Erik Visser - Web page: https://patentgrep.com/patent/US12002455B2 ## CPC - G06F 16/36: Creation of semantic tools, e.g. ontology or thesauri - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G06N 3/08: Learning methods - G06V 30/274: {Syntactic or semantic context, e.g. balancing} - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)} - G06F 16/906: Clustering; Classification - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G10L 15/16: using artificial neural networks - G10L 15/183: using context dependencies, e.g. language models - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A device includes a memory configured to store instructions. The device also includes one or more processors configured to execute the instructions to provide context and one or more items of interest corresponding to the context to a dependency network encoder to generate a semantic-based representation of the context. The one or more processors are also configured to provide the context to a data dependent encoder to generate a context-based representation. The one or more processors are further configured to combine the semantic-based representation and the context-based representation to generate a semantically-augmented representation of the context. ## Claims ### Claim 1 A device comprising: a memory configured to store instructions; and one or more processors configured to execute the instructions to: provide context and one or more items of interest corresponding to the context to a dependency network encoder to generate a semantic-based representation of the context; provide the context to a data dependent encoder to generate a context-based representation; and combine the semantic-based representation and the context-based representation to generate a semantically-augmented representation of the context. ### Claim 2 (depends on 1) The device of claim 1, wherein the one or more processors are further configured to receive sensor data from one or more sensors and generate the context based on the sensor data. ### Claim 3 (depends on 2) The device of claim 2, wherein the one or more sensors include a camera, wherein the sensor data includes image data from the camera, and wherein the one or more processors are configured to generate the context based at least in part on performing, on the image data, object recognition, emotion recognition, location recognition, person recognition, or a combination thereof. ### Claim 4 (depends on 2) The device of claim 2, wherein the one or more sensors include a microphone, wherein the sensor data includes audio data from the microphone, and wherein the one or more processors are configured to generate the context based at least in part on performing, on the audio data, speech recognition, emotion recognition, location recognition, speaker recognition, or a combination thereof. ### Claim 5 (depends on 1) The device of claim 1, wherein the one or more items include an answer indicated by the context, and wherein the one or more processors are further configured to: provide the semantically-augmented representation of the context to a generative decoder to generate one or more predicted questions for the answer; and provide the one or more predicted questions to an entailment filter to generate one or more filtered predicted questions, wherein the one or more filtered predicted questions are included in a training set to train a question answer network. ### Claim 6 (depends on 1) The device of claim 1, wherein the one or more processors are further configured to provide the semantically-augmented representation of the context to a generative decoder to generate one or more related items. ### Claim 7 (depends on 6) The device of claim 6, wherein the one or more items of interest include an answer indicated by the context, and wherein the one or more related items include a predicted question for the answer. ### Claim 8 (depends on 6) The device of claim 6, wherein the one or more items of interest include a question related to the context, and wherein the one or more related items include a predicted answer to the question. ### Claim 9 (depends on 6) The device of claim 6, wherein the one or more processors are configured to provide the one or more related items to an entailment filter to generate one or more filtered related items. ### Claim 10 (depends on 1) The device of claim 1, wherein the dependency network encoder is further configured to: provide the context to a dependency parser to generate a dependency parsing graph; generate node embeddings based at least in part on elements of the context corresponding to nodes of the dependency parsing graph; and provide an input embedding to one or more encoding layers to generate the semantic-based representation of the context, the input embedding based at least in part on the node embeddings. ### Claim 11 (depends on 10) The device of claim 10, wherein the input embedding indicates, for each node embedding of the node embeddings, parts of speech of one or more elements of the context that correspond to the node embedding. ### Claim 12 (depends on 10) The device of claim 10, wherein the input embedding indicates, for each node embedding of the node embeddings, whether one or more elements of the context that correspond to the node embedding are associated with the one or more items of interest. ### Claim 13 (depends on 10) The device of claim 10, wherein an encoding layer of the one or more encoding layers includes an attention layer and a feed forward layer, wherein the attention layer includes a multi-head attention, wherein an input of the attention layer is based on the input embedding, and wherein the semantic-based representation of the context is based on an output of the attention layer. ### Claim 14 (depends on 10) The device of claim 10, wherein the dependency network encoder further comprises a prediction layer, wherein an input of the prediction layer is based on the semantic-based representation of the context, and wherein an output of the prediction layer indicates one or more elements of the context that are predicted to be related to the one or more items of interest. ### Claim 15 (depends on 10) The device of claim 10, wherein the dependency parser is configured to generate the dependency parsing graph based at least in part on a previous context. ### Claim 16 (depends on 10) The device of claim 10, wherein the one or more encoding layers are configured to generate the semantic-based representation of the context based at least in part on a second input embedding, wherein the second input embedding is associated with a second dependency parsing graph that is based on a previous context. ### Claim 17 (depends on 10) The device of claim 10, wherein the one or more encoding layers are configured to generate the semantic-based representation of the context based at least in part on a second semantic-based representation of a previous context. ### Claim 18 (depends on 1) The device of claim 1, wherein the data dependent encoder is configured to: generate an input embedding based on elements of the context; and provide the input embedding to one or more encoding layers to generate the context-based representation. ### Claim 19 (depends on 18) The device of claim 18, wherein an encoding layer of the one or more encoding layers includes an attention layer and a feed forward layer, wherein the attention layer includes a multi-head attention, wherein an input of the attention layer is based on the input embedding, and wherein the context-based representation is based on an output of the attention layer. ### Claim 20 (depends on 18) The device of claim 18, wherein the one or more encoding layers are configured to generate the context-based representation based at least in part on a second input embedding associated with a previous context. ### Claim 21 (depends on 18) The device of claim 18, wherein the one or more encoding layers are configured to generate the context-based representation based at least in part on a second context-based representation of a previous context. ### Claim 22 A method comprising: providing, at a device, context and one or more items of interest corresponding to the context to a dependency network encoder to generate a semantic-based representation of the context; providing, at the device, the context to a data dependent encoder to generate a context-based representation; and combining, at the device, the semantic-based representation and the context-based representation to generate a semantically-augmented representation of the context. ### Claim 23 (depends on 22) The method of claim 22, further comprising: receiving, at the device, sensor data from one or more sensors; and generating the context based on the sensor data. ### Claim 24 (depends on 22) The method of claim 22, further comprising: providing, at the device, the semantically-augmented representation of the context to a generative decoder to generate one or more predicted questions for an answer that is indicated by the context, wherein the one or more items include the answer; and providing, at the device, the one or more predicted questions to an entailment filter to generate one or more filtered predicted questions, wherein the one or more filtered predicted questions are included in a training set to train a question answer network. ### Claim 25 A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: provide context and one or more items of interest corresponding to the context to a dependency network encoder to generate a semantic-based representation of the context; provide the context to a data dependent encoder to generate a context-based representation; and combine the semantic-based representation and the context-based representation to generate a semantically-augmented representation of the context. ### Claim 26 (depends on 25) The non-transitory computer-readable medium of claim 25, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to provide the semantically-augmented representation of the context to a generative decoder to generate one or more related items. ### Claim 27 (depends on 26) The non-transitory computer-readable medium of claim 26, wherein the one or more items of interest include an answer indicated by the context, and wherein the one or more related items include a predicted question for the answer. ### Claim 28 (depends on 26) The non-transitory computer-readable medium of claim 26, wherein the one or more items of interest include a question related to the context, and wherein the one or more related items include a predicted answer to the question. ### Claim 29 An apparatus comprising: means for providing context and one or more items of interest corresponding to the context to a dependency network encoder to generate a semantic-based representation of the context; means for providing the context to a data dependent encoder to generate a context-based representation; and means for combining the semantic-based representation and the context-based representation to generate a semantically-augmented representation of the context. ### Claim 30 (depends on 29) The apparatus of claim 29, wherein the means for providing the context and the one or more items of interest to the dependency network encoder, the means for providing the context to the data dependent encoder, and the means for combining are integrated into at least one of a voice-activated device, a wireless speaker and voice activated device, a portable electronic device, a car, a vehicle, a computing device, a communication device, an internet-of-things (IoT) device, a virtual reality (VR) device, a mixed reality (MR) device, an augmented reality (AR) device, or a combination thereof.