# US 2025/0094477 A1 · Natural language inferencing and deep learning-based grouping of unlabeled text

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094477 A1
- Document ID: US20250094477A1
- Status: Published application
- Kind: A1
- Application number: 18/964,875
- Filed: 2024-12-02
- Published: 2025-03-20
- Earliest claimed: 2022-12-22
- Assignees: Nasdaq, Inc.
- Applicants: Nasdaq, Inc.
- Inventors: Hyunsoo JEONG; Yongheng ZHANG; Yu-Hsuan JOU
- Web page: https://patentgrep.com/patent/US20250094477A1

## CPC

- G06F 16/353: into predefined classes
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 20/20: Ensemble learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods that provide improved (e.g., higher accuracy and time-efficient) aggregation of report data are described. Example embodiments effectively combine machine learning and deep learning in a processing pipeline to classify unlabeled source data to determine a reliable classification, or a labeling, of that source data. The classified or labeled source data can be used to generate reports that provide extensive insight into the aggregated source data.

## Claims

### Claim 1

A system comprising:
at least one memory comprising:
a database of reports, each report comprising a plurality of report sections, wherein each report section of the plurality of report sections comprises at least one question and a corresponding response;
a machine-learned model; and
a deep-learning model; and
at least one processor configured to perform operations comprising:
receiving as input a plurality of predetermined report section titles and a list of questions extracted from the database of reports;
generating, using the machine-learned model and the received input, an intermediate set of labeled questions by:
assigning, to each question in the list of questions, a respective probability score associated with each title in the plurality of predetermined report section titles; and
selecting, for each question in the list of questions, an associated predetermined title based on the assigned probability scores, wherein each labeled question in the intermediate set of labeled questions comprises a question from the list of questions and the selected predetermined title associated with the question;
selecting at least a portion of the generated intermediate set of labeled questions to be provided as input to the deep learning model;
using the selected at least portion of the generated intermediate set of labeled questions as input to the deep learning model,
repeatedly processing respective labeled questions in the deep learning model until an accuracy threshold is satisfied for the respective labeled questions;
generating a final set of labeled questions, wherein each labeled question in the final set satisfies the accuracy threshold; and
outputting aggregated labeled questions of the final set of labeled questions based on the predetermined report section titles.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the selecting at least a portion of the generated intermediate set of labeled questions to be provided as input to the deep learning model, comprises:
dynamically adapting which of the respective labeled questions are included in the selected at least portion by changing a threshold value for selecting the respective labeled questions.

### Claim 3 (depends on 1)

The system according to claim 1, wherein the repeatedly processing comprises repeatedly providing a question from the set of intermediate results as input to a neural network configured in accordance with the deep learning model.

### Claim 4 (depends on 1)

The system according to claim 1, wherein the generating, using the machine-learned model and the received input, an intermediate set of labeled questions comprises using zero shot learning (ZSL) to generate the intermediate set of labeled questions.

### Claim 5 (depends on 4)

The system according to claim 4, wherein the using at least a portion of the generated intermediate set of labeled questions as input to the deep learning model, repeatedly processing respective labeled questions until an accuracy threshold is satisfied for respective labeled questions comprises using Bidirectional Encoder Representations from Transformers (BERT) in association with the deep learning model during the repeatedly processing.

### Claim 6 (depends on 1)

The system according to claim 1, wherein the repeatedly processing respective labeled questions until an accuracy threshold is satisfied for respective labeled questions comprises regenerating the respective labeled questions by the deep learning model.

### Claim 7 (depends on 6)

The system according to claim 6, wherein the repeatedly processing respective labeled questions until an accuracy threshold is satisfied for respective labeled questions further comprises comparing the regenerated respective labeled questions to the accuracy threshold.

### Claim 8 (depends on 7)

The system according to claim 7, wherein the comparing is performed by a human operator.

### Claim 9 (depends on 1)

The system according to claim 1, wherein the outputting aggregated labeled questions of the final set of labeled questions comprises at least one of (1) determining comparative performance metrics for respective ones of the predetermined report section titles for a particular entity and for all entities, or (2) determining comparative performance metrics for respective ones of the predetermined report section titles for a particular entity for two or more time periods.

### Claim 10 (depends on 1)

The system according to claim 1, wherein the operations further comprise preprocessing the list of questions, wherein the list of questions extracted from the database of reports comprises a plurality of lines of text, wherein each line of text includes at least a report section title and a question, each of which is extracted from a report, and wherein the preprocessing comprises removing the report section title from said each line of text.

### Claim 11 (depends on 1)

The system according to claim 1, wherein the generated labeled questions each comprises a predefined report section title concatenated with a question from the list of the questions, wherein the predefined report section title is one of the plurality of predetermined report section titles.

### Claim 12 (depends on 1)

The system according to claim 1, wherein the generating, using the machine-learned model and the received input, an intermediate set of labeled questions comprises:
sequence embedding for respective questions from the list of questions;
performing classification for respective sequence embedded questions;
generating probabilities for respective sequence classified questions; and
selecting the intermediate set of labeled questions from sequence classified questions in accordance to the associated generated probabilities.

### Claim 13

A method comprising:
receiving as input, by at least one processor, a plurality of predetermined report section titles and a list of questions extracted from a database of reports stored in at least one memory, wherein in the database of reports, each report comprising a plurality of report sections, wherein each report section of the plurality of report sections comprises at least one question and a corresponding response;
generating, by the at least one processor and using a machine-learned model and the received input, an intermediate set of labeled questions by:
assigning, to each question in the list of questions, a respective probability score associated with each title in the plurality of predetermined report section titles; and
selecting, for each question in the list of questions, an associated predetermined title based on the assigned probability scores, wherein each labeled question in the intermediate set of labeled questions comprises a question from the list of questions and the selected predetermined title associated with the question;
selecting, by the at least one processor, at least a portion of the generated intermediate set of labeled questions to be provided as input to the deep learning model;
using the selected at least portion of the generated intermediate set of labeled questions as input to a deep learning model, repeatedly processing respective labeled questions by the at least one processor in the deep learning model until an accuracy threshold is satisfied for the respective labeled questions;
generating a final set of labeled questions by the at least one processor, wherein each labeled question in the final set satisfies the accuracy threshold; and
outputting, by the at least one processor, aggregated labeled questions of the final set of labeled questions based on the predetermined report section titles.

### Claim 14 (depends on 13)

The method according to claim 13, wherein the selecting at least a portion of the generated intermediate set of labeled questions to be provided as input to the deep learning model, comprises:
dynamically adapting which of the respective labeled questions are included in the selected at least portion by changing a threshold value for selecting the respective labeled questions.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the dynamically adapting is performed in accordance with a probability score associated with each of the labeled questions of the intermediate set of labeled questions.

### Claim 16 (depends on 15)

The method according to claim 15, wherein the repeatedly processing comprises repeatedly providing a question from the set of intermediate results as input to a neural network configured in accordance with the deep learning model.

### Claim 17

A non-transitory computer readable storage medium storing instructions that, when executed by at least one processor of a computer system, causes the computer system to perform operations comprising:
receiving as input a plurality of predetermined report section titles and a list of questions extracted from a database of reports stored in a memory, wherein in the database of reports, each report comprising a plurality of report sections, wherein each report section of the plurality of report sections comprises at least one question and a corresponding response;
generating, using a machine-learned model and the received input, an intermediate set of labeled questions by:
assigning, to each question in the list of questions, a respective probability score associated with each title in the plurality of predetermined report section titles; and
selecting, for each question in the list of questions, an associated predetermined title based on the assigned probability scores, wherein each labeled question in the intermediate set of labeled questions comprises a question from the list of questions and the selected predetermined title associated with the question;
selecting at least a portion of the generated intermediate set of labeled questions to be provided as input to the deep learning model;
using the selected at least portion of the generated intermediate set of labeled questions as input to a deep learning model, repeatedly processing respective labeled questions in the deep learning model until an accuracy threshold is satisfied for the respective labeled questions;
generating a final set of labeled questions, wherein each labeled question in the final set satisfies the accuracy threshold; and
outputting aggregated labeled questions of the final set of labeled questions based on the predetermined report section titles.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium according to claim 17, wherein the selecting at least a portion of the generated intermediate set of labeled questions to be provided as input to the deep learning model, comprises:
dynamically adapting which of the respective labeled questions are included in the selected at least portion by changing a threshold value for selecting the respective labeled questions.
