# US 12,182,509 B2 · Processing large-scale textual inputs using neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,509 B2 - Document ID: US12182509B2 - Status: Granted - Kind: B2 - Application number: 17/336,093 - Filed: 2021-06-01 - Published: 2021-12-02 - Granted: 2024-12-31 - Earliest claimed: 2020-06-01 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Karthik Raman; Liu Yang; Mike Bendersky; Jiecao Chen; Marc Alexander Najork - Web page: https://patentgrep.com/patent/US12182509B2 ## CPC - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/096: Transfer learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a tuple of respective input sequences to generate an output. In one aspect, one of the systems includes a neural network comprising a plurality of encoder neural networks and a head neural network, each encoder neural network configured to: receive a respective input sequence from the tuple; process the respective input sequence using one or more encoder network layers to generate an encoded representation comprising a sequence of tokens; and process each of some or all of the tokens in the sequence of tokens using a projection layer to generate a lower-dimensional representation, and the head neural network configured to: receive lower-dimensional representations of a respective proper subset of the sequence of tokens generated by the encoder neural network; and process the lower-dimensional representations to generate the output. ## Claims ### Claim 1 A system for performing a machine learning task on a tuple of a plurality of input sequences to generate an output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform one or more operations to implement: a neural network configured to perform the machine learning task, the neural network comprising (i) a plurality of encoder neural networks that comprise a different encoder neural network corresponding to each different input sequence included in the tuple and (ii) a head neural network, wherein each encoder neural network comprises one or more encoder network layers and a projection layer, wherein each encoder neural network is configured to: receive an input sequence that corresponds to the encoder neural network from the tuple, wherein the input sequence has a respective network input at each of a plurality of input positions in an input order; process the input sequence using the one or more encoder network layers to generate an encoded representation of the respective input sequence, the encoded representation comprising a sequence of tokens; determine a respective proper subset of the sequence of tokens in the encoded representation based on respective positions of the tokens in the sequence and a length of the input sequence that corresponds to the encoder neural network; and process each token in the respective proper subset using the projection layer to generate a lower-dimensional representation of the token, and wherein the head neural network is configured to: receive, from each of the plurality of encoder neural networks, the lower-dimensional representation of each token in the respective proper subset that is generated by the encoder neural network; and process the lower-dimensional representations to generate the output. ### Claim 2 (depends on 1) The system of claim 1, wherein the head neural network is further configured to access the lower-dimensional representations of the respective proper subsets of the sequences of tokens generated by the encoder neural networks from a memory. ### Claim 3 (depends on 1) The system of claim 1, wherein the lower-dimensional representations of the tokens generated by different projection layers have different dimensions from each other. ### Claim 4 (depends on 1) The system of claim 1, wherein the sequence of tokens generated by the encoder neural network comprises a corresponding token for each network input in the input sequence. ### Claim 5 (depends on 1) The system of claim 1, wherein the respective proper subset of the sequence of tokens comprises first N tokens in the sequence of tokens, and wherein N is a predetermined positive integer. ### Claim 6 (depends on 1) The system of claim 1, wherein the machine learning task is a semantic text matching task. ### Claim 7 (depends on 1) The system of claim 1, wherein the one or more encoder network layers comprise an attention layer that is configured to: receive an input sequence for the layer comprising a respective layer input at each of one or more positions; and generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions. ### Claim 8 (depends on 1) The system of claim 1, wherein the respective proper subsets of the sequences of tokens for different input sequences included in the tuple have different numbers of tokens than each other. ### Claim 9 A computer-implemented method comprising: receiving, by a neural network, a tuple of a plurality of input sequences, wherein the neural network comprises (i) a plurality of encoder neural networks that comprise a different encoder neural network corresponding to each different input sequence included in the tuple and (ii) a head neural network, wherein each encoder neural network comprises one or more encoder network layers and a projection layer; receiving, at each encoder neural network, an input sequence that corresponds to the encoder neural network from the tuple, wherein the input sequence has a respective network input at each of a plurality of input positions in an input order; processing, using the one or more encoder network layers of each encoder neural network, the input sequence to generate an encoded representation of the input sequence, the encoded representation comprising a sequence of tokens; determining a respective proper subset of the sequence of tokens in the encoded representation based on respective positions of the tokens in the sequence and a length of the input sequence that corresponds to the encoder neural network; processing, using the projection layer of each encoder neural network, each token in the respective proper subset to generate a lower-dimensional representation of the token; receiving, at the head neural network and from each of the plurality of encoder neural networks, the lower-dimensional representation of each token in the respective proper subset that is generated by the encoder neural network; and processing, using the head neural network, the lower-dimensional representations to generate an output. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein the head neural network is further configured to access the lower-dimensional representations of the respective proper subsets of the sequences of tokens generated by the encoder neural networks from a memory. ### Claim 11 (depends on 9) The computer-implemented method of claim 9, wherein the lower-dimensional representations of the tokens generated by different projection layers have different dimensions from each other. ### Claim 12 (depends on 9) The computer-implemented method of claim 9, wherein the sequence of tokens generated by the encoder neural network comprises a corresponding token for each network input in the input sequence. ### Claim 13 (depends on 9) The computer-implemented method of claim 9, wherein the respective proper subset of the sequence of tokens comprises first N tokens in the sequence of tokens, and wherein N is a predetermined positive integer. ### Claim 14 (depends on 9) The computer-implemented method of claim 9, wherein the one or more encoder network layers comprise an attention layer that is configured to: receive an input sequence for the layer comprising a respective layer input at each of one or more positions; and generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions. ### Claim 15 (depends on 9) The computer-implemented method of claim 9, further comprising training the plurality of encoder neural networks and the head neural network including initializing parameter values of the one or more encoder network layers of each encoder neural network with a predetermined set of parameter values. ### Claim 16 (depends on 15) The computer-implemented method of claim 15, wherein the training further comprises: receiving a training tuple; processing the training tuple using a trained neural network to generate a teacher network output; and training the neural network using the teacher network output generated by the trained neural network, wherein the training comprises adjusting only parameter values of the projection layers of the encoder neural networks and parameter values of the head neural network. ### Claim 17 (depends on 16) The computer-implemented method of claim 16, wherein the training further comprises: receiving another training tuple; processing the training tuple using the trained neural network to generate another teacher network output; and training the neural network using the other teacher network output generated by the trained neural network, including adjusting parameter values of the one or more encoder network layers of the encoder neural networks. ### Claim 18 (depends on 9) The computer-implemented method of claim 9, wherein the respective proper subsets of the sequences of tokens for different input sequences included in the tuple have different numbers of tokens than each other. ### Claim 19 One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement: a neural network configured to perform a machine learning task on a tuple of a plurality of input sequences to generate a network output, the neural network comprising (i) a plurality of encoder neural networks that comprise a different encoder neural network corresponding to each different input sequence included in the tuple and (ii) a head neural network, wherein each encoder neural network comprises one or more encoder network layers and a projection layer, wherein each encoder neural network is configured to: receive an input sequence that corresponds to the encoder neural network from the tuple, wherein the input sequence has a respective network input at each of a plurality of input positions in an input order; process the input sequence using the one or more encoder network layers to generate an encoded representation of the respective input sequence, the encoded representation comprising a sequence of tokens; determine a respective proper subset of the sequence of tokens in the encoded representation based on respective positions of the tokens in the sequence and a length of the input sequence that corresponds to the encoder neural network; and process each token in the respective proper subset using the projection layer to generate a lower-dimensional representation of the token, and wherein the head neural network is configured to: receive, from each of the plurality of encoder neural networks, the lower-dimensional representation of each token in the respective proper subset that is generated by the encoder neural network; and process the lower-dimensional representations to generate the output. ### Claim 20 (depends on 19) The one or more non-transitory computer storage media of claim 19, wherein the respective proper subsets of the sequences of tokens for different input sequences included in the tuple have different numbers of tokens than each other.