# US 12,217,001 B2 · Natural language processing techniques using multi-context self-attention machine learning frameworks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,001 B2 - Document ID: US12217001B2 - Status: Granted - Kind: B2 - Application number: 17/733,522 - Filed: 2022-04-29 - Published: 2023-09-28 - Granted: 2025-02-04 - Earliest claimed: 2022-02-25 - Assignees: Optum Services (Ireland) Limited - Applicants: Optum Services (Ireland) Limited - Inventors: Mostafa Bayomi; Ahmed Selim; Kieran O'Donoghue; Michael Bridges - Web page: https://patentgrep.com/patent/US12217001B2 ## CPC - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06F 40/30: Semantic analysis - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/09: Supervised learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Various embodiments of the present invention provide methods, apparatus, systems, computing devices, computing entities, and/or the like for performing natural language processing operations using a multi-context convolutional self-attention machine learning framework that comprises a shared token embedding machine learning model, a plurality of context-specific self-attention machine learning models, and a cross-context representation inference machine learning model, where each context-specific self-attention machine learning model is configured to generate, for each input text token of an input text sequence, a context-specific token representation using a context-specific self-attention mechanism that is associated with the respective distinct context window size for the context-specific self-attention machine learning model. ## Claims ### Claim 1 A computer-implemented method comprising: generating, by one or more processors and using a multi-context convolutional self-attention machine learning framework, a cross-context token representation based at least in part on an input text token of an input text sequence wherein: the multi-context convolutional self-attention machine learning framework comprises a shared token embedding machine learning model, a plurality of context-specific self-attention machine learning models, and a cross-context representation inference machine learning model, the shared token embedding machine learning model is configured to generate an initial token embedding for the input text token, a context-specific self-attention machine learning model of the plurality of context-specific self-attention machine learning models is (a) associated with a distinct context window size of a plurality of distinct context window sizes, and (b) configured to generate a context-specific token representation for the input text token based at least in part on the initial token embedding, and the cross-context representation inference machine learning model is configured to generate the cross-context token representation based at least in part on the context-specific token representation; generating, by the one or more processors and a natural language processing machine learning model, a natural language processing output for the input text sequence based at least in part on the cross-context token representation; and initiating, by the one or more processors, the performance of one or more prediction-based actions based at least in part on the natural language processing output. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein: the context-specific self-attention machine learning model comprises a Longformer machine learning model associated with a sliding window size, and the distinct context window size of the context-specific self-attention machine learning model is determined based at least in part on the sliding window size for the Longformer machine learning model. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein: the plurality of context-specific self-attention machine learning models comprises one or more attention-based convolutional neural networks (CNNs) that are respectively associated with a one or more filter widths, and a distinct window size of an attention-based CNN of the one or more attention-based CNNs is determined based at least in part on a filter width of the one or more filter widths that corresponds to the attention-based CNN. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein: the attention-based CNN comprises a pre-convolution attention-based CNN, and the pre-convolution attention-based CNN receives a convolutional input representation associated with the filter width corresponding to the attention-based CNN. ### Claim 5 (depends on 3) The computer-implemented method of claim 3, wherein: the attention-based CNN comprises a post-convolution attention-based CNN, and the post-convolution attention-based CNN generates a convolutional representation that is transformed to an attention-based average pooling representation for the input text sequence. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein: the context-specific self-attention machine learning model comprises a hierarchical convolutional attention network machine learning model associated with a window size, and the distinct context window size of the context-specific self-attention machine learning model is determined based at least in part on the respective window size for the hierarchical convolutional attention network machine learning model. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein generating the multi-context convolutional self-attention machine learning framework comprises: given C distinct context window sizes and M window-adaptive self-attention machine learning model architectures, training C*M candidate multi-context self-attention machine learning frameworks, wherein a candidate multi-context self-attention machine learning framework of the C*M candidate multi-context self-attention machine learning frameworks is associated with a unique assignment of the C distinct context window sizes to the M window-adaptive self-attention machine learning model architectures; generating a validation score for the candidate multi-context self-attention machine learning framework; and generating the multi-context convolutional self-attention machine learning framework based at least in part on the validation score. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein a window-adaptive self-attention machine learning model architecture of the M window-adaptive self-attention machine learning model architectures comprises at least one of a Longformer machine learning model architecture, a pre-convolution attention-based CNN architecture, a post-convolution attention-based CNN architecture, or a hierarchical convolutional attention network machine learning model architecture. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the cross-context representation inference machine learning model is configured to: generate a convolutional representation of the input text token based at least in part on applying a convolutional operation on the context-specific token representation for the input text token; and generate the cross-context token representation for the input text token based at least in part on applying a pooling operation on the convolutional representation. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the cross-context representation inference machine learning model is configured to: generate a sequence representation for the input text token using an attention-based machine learning model and based at least in part on the context-specific token representation for the input text token; and generate the cross-context token representation for the input text token based at least in part on the sequence representation. ### Claim 11 A system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to: generate, using a multi-context convolutional self-attention machine learning framework, a cross-context token representation based at least in part on an input text token of an input text sequence, wherein: the multi-context convolutional self-attention machine learning framework comprises a shared token embedding machine learning model, a plurality of context-specific self-attention machine learning models, and a cross-context representation inference machine learning model, the shared token embedding machine learning model is configured to generate an initial token embedding for the input text token, a context-specific self-attention machine learning model of the plurality of context-specific self-attention machine learning models is (a) associated with a distinct context window size of a plurality of distinct context window sizes, and (b) configured to generate a context-specific token representation for the input text token based at least in part on the initial token embedding, and the cross-context representation inference machine learning model is configured to generate the cross-context token representation based at least in part on the context-specific token representation; generate, using a natural language processing machine learning model, a natural language processing output for the input text sequence based at least in part on the cross-context token representation; and initiate the performance of one or more prediction-based actions based at least in part on the natural language processing output. ### Claim 12 (depends on 11) The system of claim 11, wherein: the context-specific self-attention machine learning model comprises a Longformer machine learning model associated with a sliding window size, and the distinct context window size of the context-specific self-attention machine learning model is determined based at least in part on the sliding window size for the Longformer machine learning model. ### Claim 13 (depends on 11) The system of claim 11, wherein: the plurality of context-specific self-attention machine learning models comprises one or more attention-based convolutional neural networks (CNNs) that are respectively associated with a one or more filter widths, and a distinct context window size of an attention-based CNN of the one or more attention-based CNNs is determined based at least in part on a filter width of the one or more filter widths that corresponds to the attention-based CNN. ### Claim 14 (depends on 13) The system of claim 13, wherein: the attention-based CNN comprises a pre-convolution attention-based CNN, and the pre-convolution attention-based CNN receives a convolutional input representation associated with the filter width corresponding to the attention-based CNN. ### Claim 15 (depends on 13) The system of claim 13, wherein: the attention-based CNN comprises a post-convolution attention-based CNN, and the post-convolution attention-based CNN generates a convolutional representation that is transformed to an attention-based average pooling representation for the input text sequence. ### Claim 16 (depends on 11) The system of claim 11, wherein: the context-specific self-attention machine learning model comprises a hierarchical convolutional attention network machine learning model associated with a window size, and the distinct context window size of the context-specific self-attention machine learning model is determined based at least in part on the window size for the hierarchical convolutional attention network machine learning model. ### Claim 17 (depends on 11) The system of claim 11, wherein, to generate the multi-context convolutional self-attention machine learning framework, the one or more processors are further configured to: given C distinct context window sizes and M window-adaptive self-attention machine learning model architectures, train C * M candidate multi-context self-attention machine learning frameworks, wherein a candidate multi-context self-attention machine learning framework of the C * M candidate multi-context self-attention machine learning frameworks is associated with a unique assignment of the C distinct context window sizes to the M window-adaptive self-attention machine learning model architectures; generate a validation score for the candidate multi-context self-attention machine learning framework; and generate the multi-context convolutional self-attention machine learning framework based at least in part on the validation score. ### Claim 18 (depends on 17) The system of claim 17, wherein a window-adaptive self-attention machine learning model architecture of the M window-adaptive self-attention machine learning model architectures comprises at least one of a Longformer machine learning model architecture, a pre-convolution attention-based CNN architecture, a post-convolution attention-based CNN architecture, or a hierarchical convolutional attention network machine learning model architecture. ### Claim 19 (depends on 11) The system of claim 11, wherein the cross-context representation inference machine learning model is configured to: generate a convolutional representation of the input text token based at least in part on applying a convolutional operation on the context-specific token representation for the input text token; and generate the cross-context token representation for the input text token based at least in part on applying a pooling operation on the convolutional representation. ### Claim 20 One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to: generate, using a multi-context convolutional self-attention machine learning framework, a cross-context token representation based at least in part on an input text token of an input text sequence, wherein: the multi-context convolutional self-attention machine learning framework comprises a shared token embedding machine learning model, a plurality of context-specific self-attention machine learning models, and a cross-context representation inference machine learning model, the shared token embedding machine learning model is configured to generate an initial token embedding for the input text token, a context-specific self-attention machine learning model of the plurality of context-specific self-attention machine learning models is (a) associated with a distinct context window size of a plurality of distinct context window sizes, and (b) configured to generate a context-specific token representation for the input text token based at least in part on the initial token embedding, and the cross-context representation inference machine learning model is configured to generate the cross-context token representation based at least in part on the context-specific token representation; generate, using a natural language processing machine learning model, a natural language processing output for the input text sequence based at least in part on the cross-context token representation; and initiate the performance of one or more prediction-based actions based at least in part on the natural language processing output.