# US 2019/0318648 A1 · Systems and methods for interactive language acquisition with one-shot visual concept learning through a conversational game > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318648 A1 - Document ID: US20190318648A1 - Status: Published application - Kind: A1 - Application number: 15/951,905 - Filed: 2018-04-12 - Published: 2019-10-17 - Earliest claimed: 2018-04-12 - Assignees: Baidu USA LLC - Applicants: Baidu USA, LLC - Inventors: Haichao ZHANG; Haonan YU; Wei XU - Web page: https://patentgrep.com/patent/US20190318648A1 ## CPC - G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26) - G06F 40/30: Semantic analysis - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N 3/096: Transfer learning - G09B 7/04: characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question or supplying a further explanation - G09B 19/06: Foreign languages (with audible presentation of material to be studied G09B5/04) - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Described herein are systems and methods for interactive language acquisition with one-shot concept learning through a conversational environment. Supervised language learning is limited by the ability of capturing mainly the statistics of training data, and is hardly adaptive to new scenarios or flexible for acquiring new knowledge without inefficient retraining or catastrophic forgetting. In one or more embodiments, conversational interaction serves as a natural interface both for language learning and for novel knowledge acquisition. Embodiments of a joint imitation and reinforcement approach are disclosed for grounded language learning through interactive conversation. An agent trained with this approach is able to actively acquire information by asking questions about novel objects and use the just-learned knowledge in subsequent conversations in a one-shot fashion. Results compared with other methods verified the effectiveness of embodiments disclosed herein. ## Claims ### Claim 1 A computer-implemented method for interaction-based language learning in a session comprising a plurality of time steps, the method comprising: receiving, at a time step, an input comprising a visual image and a natural language input comprising zero or more words related to the visual image; using a visual encoder to encode the visual image into a visual feature representation; using an attention mechanism to encode the natural language input into a natural language input representation; generating at a first neural network a state vector based on at least the natural language input and a prior state vector; storing, via the first neural network, into a memory the visual feature representation and the natural language input representation as a visual modality and sentence modality pair; passing the state vector through a residue-structured network to produce a control vector as a state input to modulate behavior of a second neural network; and using at least the control vector to adaptively select, using learned attention and content importance, between a proposed response portion indicated by the second neural network and a proposed response portion indicated using at least a portion of a natural language input representation retrieved from the memory using the visual feature representation as a visual key to retrieve the natural language input representation to form a response corresponding to the input. ### Claim 2 (depends on 1) The computer-implemented method of claim 1 further comprising: using a final state of the second neural network at the time step as an input to the first neural network at a next time step. ### Claim 3 (depends on 1) The computer-implemented method of claim 1 wherein the session ends if the response is a correct response or when a maximum number of interaction steps is reached. ### Claim 4 (depends on 1) The computer-implemented method of claim 1 wherein the residue-structured network is an identity mapping augmented network with a learnable controller. ### Claim 5 (depends on 3) The computer-implemented method of claim 3 wherein the learnable controller receives the state vector and generates an output, the control vector is obtained by combining the controller's output with the state vector. ### Claim 6 (depends on 1) The computer-implemented method of claim 1 further comprising: generating feedback and reward, by a teacher, according to the natural language, the visual input and the generated response. ### Claim 7 (depends on 6) The computer-implemented method of claim 6 wherein the reward is a scalar-valued feedback with positive value as encouragement and negative value as discouragement according to correctness of the response. ### Claim 8 (depends on 6) The computer-implemented method of claim 6 wherein the reward is leveraged to adjust an action policy in generating a response, by maximizing expected future reward as represented by a reinforce cost function. ### Claim 9 (depends on 8) The computer-implemented method of claim 8 wherein the action policy is simultaneously optimized by minimizing the error in predicting the natural language input by an imitation cost function at the first neural network and sharing parameters between the first and second neural networks. ### Claim 10 (depends on 1) The computer-implemented method of claim 1 wherein in a memory write operation, a content importance gate is computed as a function of the natural language input to adaptively control whether to write a content of the natural language input into the memory. ### Claim 11 A computer-implemented method for an trained agent to use acquired language ability and developed one-shot learning ability, the method comprising: receiving an input comprising a visual image and a natural language input comprising zero or more words related to the visual image; encoding, using a visual encoder, the visual image into a visual key; using an attention mechanism to encode the natural language input into a natural language input representation; generating at a first neural network a state vector based on at least the natural language input and a prior state vector; producing a control vector based on the state vector to modulate behavior of a second neural network; and using a fusion gate that is at least partially controlled by the control vector to adaptively select between a proposed response portion indicated by the second neural network and a proposed response portion indicated using at least a portion of a natural language input representation retrieved from a memory using the visual key to retrieve the natural language input representation to form a response corresponding to the input, the memory stores at least one visual modality and sentence modality pair, each pair comprising a stored visual feature representation and a stored natural language input representation. ### Claim 12 (depends on 11) The computer-implemented method of claim 11 wherein the at least one visual modality and sentence modality pair is stored, via the first neural network, using historic natural language inputs and associated visual inputs based on learned content importance. ### Claim 13 (depends on 12) The computer-implemented method of claim 12 wherein the stored natural language input representation is obtained based on a matrix with columns being embedding vectors of all words from a related historic natural language input and a normalized attention vector representing the relative importance of each word in the related historic natural language input. ### Claim 14 (depends on 11) The computer-implemented method of claim 11 wherein the proposed response portion indicated by the second neural network is extracted from the natural language input via word-level attention and content importance jointly. ### Claim 15 (depends on 11) The computer-implemented method of claim 11 wherein the natural language input representation is retrieved from the memory by cross modality retrieval, which is implemented by using the visual key to obtain a soft reading weight. ### Claim 16 (depends on 15) The computer-implemented method of claim 15 wherein the soft reading weight is obtained by calculating a cosine similarity between the visual key and stored visual feature representation, the at least a portion of the natural language input representation is retrieved as an embedding vector corresponding to a label word associated with the visual image by implementing a product between the obtained soft reading weight and the stored natural language input representation. ### Claim 17 A system for interactive language acquisition with one-shot visual concept learning, the system comprising: a visual encoder to encode a visual image into a visual feature representation; a first neural network to generate a state vector based on at least a natural language input comprising zero or more words related to the visual image and a prior state vector using historical information; a memory storing at least one visual modality and sentence modality pair, each modality and sentence modality pair comprising a visual feature representation and a natural language input representation; a residue-structured network receiving the state vector to produce a control vector; and a second neural network receiving the control vector for behavior modulation, the second neural network adaptively selects between a proposed response portion indicated by the second neural network and a proposed response portion indicated using at least a portion of a natural language input representation retrieved from the memory using the visual feature representation as a visual key to retrieve the natural language input representation to form a response corresponding to the input. ### Claim 18 (depends on 17) The system of claim 17 wherein the first neural network and the second neural network are both recurrent neural networks (RNNs) and share parameters. ### Claim 19 (depends on 17) The system of claim 17 wherein the residue-structured network comprises a learnable controller to receive the state vector and generate an output, the control vector is obtained by combining the controller's output to the state vector. ### Claim 20 (depends on 17) The system of claim 17 wherein the second neural network comprises one or more fusion gates to adaptively switch between the proposed response portion indicated by the second neural network and the proposed response portion retrieved from the memory based, at least in part, upon learned attention and content importance.