Methods of automatically generating formatted annotations of doctor-patient conversations

A data processing system, including:

• at least one processing device;
• and
• a memory storing instructions that, when executed by the at least one processing device, cause the at least one processing device to perform operations including:
• accessing a digital resource that includes a plurality of sections;
• accessing, from a hardware storage device, a hierarchical classifier configured to detect one or more portions of a communication with increased likelihood of being cited as evidence associated with a particular one of the sections, relative to a likelihood of one or more portions of another communication being cited as the evidence for the particular one of the sections;
• receiving, from one or more data sources, a stream of data representing a communication;
• for each section of the plurality of sections:
• extracting, from the stream of data by a first portion of the hierarchical classifier, utterances from the communication by:
• identifying a diagnosis or symptom relevant to the section as defined in a medical-entity-matching baseline;
• identifying, based on section embedding data associated with the section, a portion of the communication previously used as evidence for one or more natural language summaries;
• or
• identifying, based on section embedding data associated with the section, a portion of the communication associated with a ground truth label of the hierarchical classifier;
• clustering an utterance that is extracted from the communication with similar utterances into a cluster when a proximity of the utterance is less than a threshold distance in the communication from the similar utterances, the threshold distance tuned to cause a first number of clusters for the communication to approximate a second number of clusters in ground truth data, wherein each cluster forms a sentence in a natural language summary of content of the cluster;
• repeating the clustering until the communication is divided into a set of proximity-based clusters including the cluster,
• sorting the cluster among the clusters of the section, the sorting based on an earliest respective utterance in each of the clusters;
• for each sorted cluster of the section, applying a second portion of the hierarchical classifier to utterances of the cluster, the second portion of the hierarchical classifier including a multi-label classifier that generates an output representing an increased likelihood of each utterance of the cluster being cited as evidence for the section in the natural language summary of the communication, the multi-label classifier being conditioned based on the section that is selected each utterance input to the multi-label classifier being contextualized with other utterances of the cluster;
• based on applying the multi-label classifier to the utterances in the cluster, generating a natural language summary of content in the cluster;
• and
• repeating application of the multi-label classifier to each of the proximity-based clusters for the section to generate the natural language summary of the section;
• and
• combining the natural language summary for each section into a structured record.

One or more non-transitory computer-readable media storing instructions that, when executed by at least one processing device, cause the at least one processing device to perform operations including:

• accessing a digital resource that includes a plurality of sections;
• accessing, from a hardware storage device, a hierarchical classifier configured to detect one or more portions of a communication with increased likelihood of being cited as evidence associated with a particular one of the sections, relative to a likelihood of one or more portions of another communication being cited as the evidence for the particular one of the sections;
• receiving, from one or more data sources, a stream of data representing a communication;
• for each section of the plurality of sections:
• extracting, from the stream of data by a first portion of the hierarchical classifier, utterances from the communication by:
• identifying a diagnosis or symptom relevant to the section as defined in a medical-entity-matching baseline;
• identifying, based on section embedding data associated with the section, a portion of the communication previously used as evidence for one or more natural language summaries;
• or
• identifying, based on section embedding data associated with the section, a portion of the communication associated with a ground truth label of the hierarchical classifier;
• clustering an utterance that is extracted from the communication with similar utterances into a cluster when a proximity of the utterance is less than a threshold distance in the communication from the similar utterances, the threshold distance tuned based on ground truth data to cause a first number of clusters for the communication to approximate a second number of clusters in the ground truth data, wherein each cluster forms a sentence in a natural language summary of content of the cluster;
• repeating the clustering until the communication is divided into a set of proximity-based clusters including the cluster,
• sorting the cluster among the clusters of the section, the sorting based on an earliest respective utterance in each of the clusters;
• for each sorted cluster of the section, applying a second portion of the hierarchical classifier to utterances of the cluster, the second portion of the hierarchical classifier including a multi-label classifier that generates an output representing an increased likelihood of each utterance of the cluster being cited as evidence for the section in the natural language summary of the communication, the multi-label classifier being conditioned based on the section that is selected each utterance input to the multi-label classifier being contextualized with other utterances of the cluster;
• based on applying the multi-label classifier to the utterances in the cluster, generating a natural language summary of content in the cluster;
• and
• repeating application of the multi-label classifier to each of the proximity-based clusters for the section to generate the natural language summary of the section;
• and
• combining the natural language summary for each section into a structured record.