# US 2021/0241135 A1 · Method and server for managing a dataset in the context of artificial intelligence > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241135 A1 - Document ID: US20210241135A1 - Status: Published application - Kind: A1 - Application number: 16/779,481 - Filed: 2020-01-31 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: ServiceNow Canada Inc. - Applicants: Element AI Inc. - Inventors: Frédéric BRANCHAUD-CHARRON; Parmida ATIGHEHCHIAN; Jan FREYBERG; Lorne SCHELL - Web page: https://patentgrep.com/patent/US20210241135A1 ## CPC - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 5/04: Inference or reasoning models - G06F 16/285: {Clustering or classification} - G06N 3/063: using electronic means - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method and system for managing a dataset. An artificial intelligence (AI) model is to be used on the dataset. A data mask describes a labeling status of the data items. A loop is repeated until patience parameters are satisfied. The loop comprises receiving trusted labels provided by trusted labelers; updating the data mask; from a labelled data items subset, training the AI model; cloning the trained AI model into a local AI model on processing nodes; creating and chunking a randomized unlabeled subset into data subsets for dispatching to the processing nodes; receiving an indication that predicted label answers have been inferred by the processing nodes using the local AI model; computing a model uncertainty measurement from statistical analysis of the predicted label answers. The patience parameters include one or more of a threshold value on the model uncertainty measurement and information gain between different training cycles. ## Claims ### Claim 1 A method for managing a dataset, the dataset comprising data items, labelling tasks associated to the data items and labels corresponding to answers to the labelling tasks, the method comprising: determining an artificial intelligence (AI) model to be used on the dataset; creating a data mask describing a labeling status of the data items of the dataset; and repeating a loop, until patience parameters are satisfied: receiving one or more trusted labels provided by one or more trusted data labelers; updating the data mask by changing the labeling status of the data items for which a trusted label is received; from a labelled data items subset obtained using the data mask, training the AI model; cloning the trained AI model into a local AI model on the processing nodes; from an unlabelled data items subset obtained using the data mask, creating a randomized unlabeled subset having fewer members than the unlabelled data items subset; at a cluster manager server, chunking the randomized unlabeled subset into a plurality of data subsets for dispatching to one or more of the processing nodes; at the cluster manager server, receiving an indication that one or more predicted label answers have been inferred by the one or more processing nodes using the local AI model; and computing a model uncertainty measurement from statistical analysis of the one or more predicted label answers; wherein the patience parameters include one or more of: a threshold value on the model uncertainty measurement and information gain between different training cycles. ### Claim 2 (depends on 1) The method of claim 1, further comprising updating the dataset by concatenating the predicted label answers received from the one or more processing nodes into an updated dataset to be used in a next iteration of the loop. ### Claim 3 (depends on 1) The method of claim 1 wherein receiving the indication further comprises receiving a local model uncertainty measurement for the local AI model from the respective one or more processing nodes. ### Claim 4 (depends on 1) The method of claim 1, further comprising receiving a computed information gain from the one or more processing nodes. ### Claim 5 (depends on 1) The method of claim 1, further comprising receiving a computed relevancy values for one or more predicted labels from the one or more processing nodes. ### Claim 6 (depends on 5) The method of claim 5, further comprising requesting trusted labels for data items having associated therewith higher relevancy value compared to other ones of the data items. ### Claim 7 (depends on 1) The method of claim 1, wherein the data mask is a vector in which each component is 1 if the components is related to a labelled data item and 0 if the components is related to an unlabelled data item. ### Claim 8 A server for managing a dataset, the dataset comprising data items, labelling tasks associated to the data items and labels corresponding to answers to the labelling tasks, the server comprising: a cluster manager; and a processing module for: determining an artificial intelligence (AI) model to be used on the dataset; creating a data mask describing a labeling status of the data items of the dataset; repeating a loop, until patience parameters are satisfied: receiving one or more trusted labels provided by one or more trusted data labelers; updating the data mask by changing the labeling status of the data items for which a trusted label is received; from a labelled data items subset obtained using the data mask, training the AI model; cloning the trained AI model into a local AI model on the processing nodes; from an unlabelled data items subset obtained using the data mask, creating a randomized unlabeled subset having fewer members than the unlabelled data items subset; chunking, using the cluster manager, the randomized unlabeled subset into a plurality of data subsets for dispatching to one or more of the processing nodes; and receiving, through the cluster manager, an indication that one or more predicted label answers have been inferred by the one or more processing nodes using the local AI model; computing a model uncertainty measurement from statistical analysis of the one or more predicted label answers; wherein the patience parameters include one or more of: a threshold value on the model uncertainty measurement and information gain between different training cycles. ### Claim 9 (depends on 8) The server of claim 8 wherein the processing module is further for updating the dataset by concatenating the predicted label answers received from the one or more processing nodes into an updated dataset to be used in a next iteration of the loop. ### Claim 10 (depends on 8) The server of claim 8 wherein the processing module receives the indication by receiving a local model uncertainty measurement for the local AI model from the respective one or more processing nodes. ### Claim 11 (depends on 8) The server of claim 8 wherein the processing module is further for receiving a computed information gain from the one or more processing nodes. ### Claim 12 (depends on 8) The server of claim 8 wherein the processing module is further for receiving a computed relevancy values for one or more predicted labels from the one or more processing nodes. ### Claim 13 (depends on 12) The server of claim 12 wherein the processing module is further for requesting trusted labels for data items having associated therewith higher relevancy value compared to other ones of the data items. ### Claim 14 (depends on 8) The server of claim 8, wherein the data mask is a vector in which each component is 1 if the components is related to a labelled data item and 0 if the components is related to an unlabelled data item.