# US 2021/0241040 A1 · Systems and Methods for Ground Truth Dataset Curation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241040 A1 - Document ID: US20210241040A1 - Status: Published application - Kind: A1 - Application number: 17/168,704 - Filed: 2021-02-05 - Published: 2021-08-05 - Earliest claimed: 2020-02-05 - Assignees: Origin Labs, Inc. - Applicants: Origin Labs, Inc. - Inventors: Darick M. Tong; Evan Szu; Clifford Szu; Nishant Borude; Nivedita Suresh - Web page: https://patentgrep.com/patent/US20210241040A1 ## CPC - G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients - G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/2413: based on distances to training or reference patterns - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/048: Activation functions - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) - G16H 30/40: for processing medical images, e.g. editing - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06N 20/20: Ensemble learning - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods of the present disclosure enable ground truth dataset creation using processing devices configured to receive at least one user profile of at least one user and user-generated labels for each sample of at least one sample, where the user-generated labels are associated with the at least one user profile. A most likely label for each sample is produced using a weighting of each user-generated label according to user attributes in each user profile associated with each user-generated label. Each most likely label is recorded for each sample in a ground-truth dataset associated with each sample to identify a ground truth of the at least one sample, and each most likely label and each sample is provided to a machine learning model to train the machine learning model according to a difference between a predicted label for each sample and each most likely label. ## Claims ### Claim 1 A method comprising: receiving, by at least one processor, user-generated labels for each sample of at least one sample; wherein the user-generated labels are associated with at least one user profile of at least one user; receiving, by the at least one processor, the at least one user profile; producing, by the at least one processor, a most likely label for each sample based at least in part on a weighting of each user-generated label of the user-generated labels according to user attributes represented in each user profile of the at least one user profile associated with each user-generated label; recording, by the at least one processor, each most likely label for each sample in a ground-truth dataset associated with each sample to identify a ground truth of the at least one sample; and providing, by the at least one processor, each most likely label and each sample to a machine learning model to train the machine learning model according to a difference between a predicted label for each sample and each most likely label. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining, by the at least one processor, a distance from a consensus of each user-generated label of the user-generated labels based at least in part on a distance measure of each user-generated label from the most likely label; and determining, by the at least one processor, a user skill level for each user profile based on the distance from the consensus of each user-generated label associated with each user profile. ### Claim 3 (depends on 1) The method of claim 1, further comprising: accessing, by the at least one processor, prior distances from consensus for each user profile, wherein the prior distances from the consensus comprises a distance measure of each prior user-generated label from each prior most likely label; determining, by the at least one processor, an aggregated distance for each user profile based on a statistical aggregation of the prior distances of each user profile; and generating, by the at least one processor, the weighting of each user-generated label of the user-generated labels according to user attributes represented in each user profile of the at least one user profile associated with each user-generated label. ### Claim 4 (depends on 3) The method of claim 3, wherein the statistical aggregation comprises an average. ### Claim 5 (depends on 3) The method of claim 3, further comprising: determining, by the at least one processor, a weighted mean for each sample according to a weighting of the user-generated labels of each sample; determining, by the at least one processor, a sampling distribution of the user-generated labels and the weighted mean for each sample; and determining, by the at least one processor, a confidence of each most likely label for each sample based on a standard deviation of the sampling distribution. ### Claim 6 (depends on 5) The method of claim 5, further comprising filtering, by the at least one processor, each likely label based on a comparison of the confidence of each most likely label to a threshold confidence. ### Claim 7 (depends on 5) The method of claim 5, wherein the confidence of each most likely label comprises a confidence interval. ### Claim 8 (depends on 5) The method of claim 5, wherein the confidence of each most likely label comprises a false positive rate. ### Claim 9 (depends on 1) The method of claim 1, wherein each sample comprises biological sample data. ### Claim 10 (depends on 9) The method of claim 9, wherein the biological sample data comprises medical imagery. ### Claim 11 A system comprising: at least one processor in communication with at least one memory configured to access instructions in the at least one memory that cause the at least one processor to perform steps to: receive user-generated labels for each medical sample of at least one sample; wherein the user-generated labels are associated with at least one user profile of at least one user; receive the at least one user profile; produce a most likely label for each medical sample based at least in part on a weighting of each user-generated label of the user-generated labels according to user attributes represented in each user profile of the at least one user profile associated with each user-generated label; and record each most likely label for each medical sample in a ground-truth dataset associated with each medical sample to identify a ground truth of the at least one sample. ### Claim 12 (depends on 11) The system of claim 11, wherein the instructions cause the at least one processor to further perform steps to: determine a distance from a consensus of each user-generated label of the user-generated labels based at least in part on a distance measure of each user-generated label from the most likely label; and determine a user skill level for each user profile based on the distance from the consensus of each user-generated label associated with each user profile. ### Claim 13 (depends on 11) The system of claim 11, wherein the instructions cause the at least one processor to further perform steps to: access prior distances from consensus for each user profile, wherein the prior distances from the consensus comprises a distance measure of each prior user-generated label from each prior most likely label; determine an aggregated distance for each user profile based on a statistical aggregation of the prior distances of each user profile; and generate the weighting of each user-generated label of the user-generated labels according to user attributes represented in each user profile of the at least one user profile associated with each user-generated label. ### Claim 14 (depends on 13) The system of claim 13, wherein the statistical aggregation comprises an average. ### Claim 15 (depends on 13) The system of claim 13, wherein the instructions cause the at least one processor to further perform steps to: determine a weighted mean for each sample according to a weighting of the user-generated labels of each sample; determine a sampling distribution of the user-generated labels and the weighted mean for each sample; and determine a confidence of each most likely label for each sample based on a standard deviation of the sampling distribution. ### Claim 16 (depends on 15) The system of claim 15, wherein the instructions cause the at least one processor to further perform steps to filter each likely label based on a comparison of the confidence of each most likely label to a threshold confidence. ### Claim 17 (depends on 15) The system of claim 15, wherein the confidence of each most likely label comprises a confidence interval. ### Claim 18 (depends on 15) The system of claim 15, wherein the confidence of each most likely label comprises a false positive rate. ### Claim 19 (depends on 11) The system of claim 11, wherein each sample comprises biological sample data. ### Claim 20 (depends on 19) The system of claim 19, wherein the biological sample data comprises medical imagery.