# US 11,907,860 B2 · Targeted data acquisition for model training > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,860 B2 - Document ID: US11907860B2 - Status: Granted - Kind: B2 - Application number: 17/935,341 - Filed: 2022-09-26 - Published: 2023-01-19 - Granted: 2024-02-20 - Earliest claimed: 2020-05-14 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Namit Kabra; Ritesh Kumar Gupta; Vijay Ekambaram; Smitkumar Narotambhai Marvaniya - Web page: https://patentgrep.com/patent/US11907860B2 ## CPC - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 5/04: Inference or reasoning models - G06N 3/045: Combinations of networks - G06N 3/0475: Generative networks - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/091: Active learning - G06N 3/094: Adversarial learning - G06N 3/096: Transfer learning - G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence - G06N 20/00: Machine learning - G06N 3/047: Probabilistic or stochastic networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Targeted acquisition of data for model training includes automatically generating metadata describing samples, of an initial dataset, in neighborhoods of an embedding space in which the samples are embedded. The samples described by the automatically generated metadata are classified by a classification model, and include both correctly classified samples in the neighborhoods and incorrectly classified samples in the neighborhoods. Additionally, attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples are identified, and queries are generated based on the identified attributes, the queries tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification. ## Claims ### Claim 1 A computer-implemented method comprising: identifying attributes of one or more correctly classified samples, of a collection of samples classified by a classification model, and one or more incorrectly classified samples of the collection of samples; and generating queries based on the identified attributes, the queries tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification. ### Claim 2 (depends on 1) The method of claim 1, wherein the samples of the collection of samples are classified by the classification model in neighborhoods or an embedding space in which the samples are embedded. ### Claim 3 (depends on 1) The method of claim 1, further comprising: obtaining an embedding of the collection of samples in an embedding space in which the samples are embedded, the embedding presenting a respective classification of each sample of the collection of samples by the classification model; and defining neighborhoods of the embedding space, wherein each neighborhood of the neighborhoods comprises a respective at least one incorrectly classified sample of the collection of samples, embedded in the embedding space, that the classification model incorrectly classified, and a respective at least one correctly classified sample of the collection of samples, embedded in the embedding space, that the classification model correctly classified. ### Claim 4 (depends on 3) The method of claim 3, wherein the defining the neighborhoods comprises using at least one radius to define a region of the embedding space centered around an incorrectly classified sample, wherein the region defines a neighborhood of the neighborhoods, and wherein any samples within the region are samples within that neighborhood. ### Claim 5 (depends on 1) The method of claim 1, further comprising constructing, using generated metadata, knowledge graphs that inform the identified attributes of the one or more correctly classified samples of the collection of samples and the one or more incorrectly classified samples of the collection of samples. ### Claim 6 (depends on 5) The method of claim 5, wherein the constructing the knowledge graphs comprises constructing a respective knowledge graph for each neighborhood of the neighborhoods based on metadata generated for that neighborhood, wherein the method further comprises combining attributes informed by at least some of the constructed knowledge graphs to construct an aggregate knowledge graph, and wherein the generating the queries generates at least one query of the generated queries from the aggregate knowledge graph. ### Claim 7 (depends on 1) The method of claim 1, wherein the collection of samples comprises images, wherein at least some of the identified attributes comprise an object depicted in at least some of the images, and wherein the generating the queries uses visibility flags, indicating whether the object is visible in the images, in determining whether to tailor at least one query to include or omit results in which the object is visible. ### Claim 8 (depends on 1) The method of claim 1, wherein the generated queries are implemented as crawl paths for crawling remote resources to retrieve the additional training data. ### Claim 9 (depends on 1) The method of claim 1, wherein at least some of the generated queries are multimodal, in which a multimodal query queries for both text and image results. ### Claim 10 (depends on 1) The method of claim 1, further comprising: ranking the generated queries into a ranked list of queries; and selecting, from the ranked list of queries, a query to issue, wherein the selecting applies a budget comprising one or more constraints on resources to execute the query, the resources comprising at least one selected from the group consisting of: computing cost, memory, time, and electrical power. ### Claim 11 (depends on 1) The method of claim 1, further comprising: issuing at least one query of the generated queries; retrieving, in response to the issuing, additional samples on which to train the classification model; and retraining the classification model using the additional samples. ### Claim 12 (depends on 11) The method of claim 11, further comprising, based on the retraining: iterating one or more times: (i) the identifying attributes, (ii) the generating queries, (iii) the issuing at least one query, (v) the retrieving additional samples, and (vi) the retraining the classification model; and checking whether to halt the iterating by testing stability of the classification model and determining whether a threshold has been reached to avoid overfitting the classification model. ### Claim 13 A computer system comprising: a memory; and a processor in communication with the memory, wherein the computer system is configured to perform a method comprising: identifying attributes of one or more correctly classified samples, of a collection of samples classified by a classification model, and one or more incorrectly classified samples of the collection of samples; and generating queries based on the identified attributes, the queries tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification. ### Claim 14 (depends on 13) The computer system of claim 13, wherein the samples of the collection of samples are classified by the classification model in neighborhoods or an embedding space in which the samples are embedded. ### Claim 15 (depends on 13) The computer system of claim 13, wherein the method further comprises: obtaining an embedding of the collection of samples in an embedding space in which the samples are embedded, the embedding presenting a respective classification of each sample of the collection of samples by the classification model; and defining neighborhoods of the embedding space, wherein each neighborhood of the neighborhoods comprises a respective at least one incorrectly classified sample of the collection of samples, embedded in the embedding space, that the classification model incorrectly classified, and a respective at least one correctly classified sample of the collection of samples, embedded in the embedding space, that the classification model correctly classified. ### Claim 16 (depends on 13) The computer system of claim 13, wherein the method further comprises constructing, using generated metadata, knowledge graphs that inform the identified attributes of the one or more correctly classified samples of the collection of samples and the one or more incorrectly classified samples of the collection of samples. ### Claim 17 (depends on 16) The computer system of claim 16, wherein the constructing the knowledge graphs comprises constructing a respective knowledge graph for each neighborhood of the neighborhoods based on metadata generated for that neighborhood, wherein the method further comprises combining attributes informed by at least some of the constructed knowledge graphs to construct an aggregate knowledge graph, and wherein the generating the queries generates at least one query of the generated queries from the aggregate knowledge graph. ### Claim 18 (depends on 13) The computer system of claim 13, wherein the collection of samples comprises images, wherein at least some of the identified attributes comprise an object depicted in at least some of the images, and wherein the generating the queries uses visibility flags, indicating whether the object is visible in the images, in determining whether to tailor at least one query to include or omit results in which the object is visible. ### Claim 19 (depends on 13) The computer system of claim 13, wherein at least some of the generated queries are multimodal, in which a multimodal query queries for both text and image results. ### Claim 20 (depends on 13) The computer system of claim 13, wherein the method further comprises: ranking the generated queries into a ranked list of queries; and selecting, from the ranked list of queries, a query to issue, wherein the selecting applies a budget comprising one or more constraints on resources to execute the query, the resources comprising at least one selected from the group consisting of: computing cost, memory, time, and electrical power. ### Claim 21 A computer program product comprising: a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising: identifying attributes of one or more correctly classified samples, of a collection of samples classified by a classification model, and one or more incorrectly classified samples of the collection of samples; and generating queries based on the identified attributes, the queries tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification. ### Claim 22 (depends on 21) The computer program product of claim 21, wherein the method further comprises: obtaining an embedding of the collection of samples in an embedding space in which the samples are embedded, the embedding presenting a respective classification of each sample of the collection of samples by the classification model; and defining neighborhoods of the embedding space, wherein each neighborhood of the neighborhoods comprises a respective at least one incorrectly classified sample of the collection of samples, embedded in the embedding space, that the classification model incorrectly classified, and a respective at least one correctly classified sample of the collection of samples, embedded in the embedding space, that the classification model correctly classified. ### Claim 23 (depends on 21) The computer program product of claim 21, wherein the method further comprises constructing, using generated metadata, knowledge graphs that inform the identified attributes of the one or more correctly classified samples of the collection of samples and the one or more incorrectly classified samples of the collection of samples. ### Claim 24 (depends on 23) The computer program product of claim 23, wherein the constructing the knowledge graphs comprises constructing a respective knowledge graph for each neighborhood of the neighborhoods based on metadata generated for that neighborhood, wherein the method further comprises combining attributes informed by at least some of the constructed knowledge graphs to construct an aggregate knowledge graph, and wherein the generating the queries generates at least one query of the generated queries from the aggregate knowledge graph. ### Claim 25 (depends on 21) The computer program product of claim 21, wherein the collection of samples comprises images, wherein at least some of the identified attributes comprise an object depicted in at least some of the images, and wherein the generating the queries uses visibility flags, indicating whether the object is visible in the images, in determining whether to tailor at least one query to include or omit results in which the object is visible.