# US 2019/0318198 A1 · Continual learning for multi modal systems using crowd sourcing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318198 A1 - Document ID: US20190318198A1 - Status: Published application - Kind: A1 - Application number: 15/992,013 - Filed: 2018-05-29 - Published: 2019-10-17 - Earliest claimed: 2018-04-13 - Assignees: Cisco Technology, Inc. - Applicants: Cisco Technology, Inc. - Inventors: Keith Griffin; Eric Chen - Web page: https://patentgrep.com/patent/US20190318198A1 ## CPC - G06N 20/00: Machine learning - G06F 18/23: Clustering techniques - G06F 18/253: {of extracted features} - G06F 18/41: {Interactive pattern learning with a human teacher} - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/987: {with the intervention of an operator} - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/06: Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice (G10L15/14 takes precedence) - G10L 15/063: {Training} - G10L 2015/0631: {Creating reference templates; Clustering} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Systems, methods, and devices are disclosed for training a model. Media data is separated into one or more clusters, each cluster based on a feature from a first model. The media data of each cluster is sampled and, based on an analysis of the sampled media data, an accuracy of the media data of each cluster is determined. The accuracy is associated with the feature from the first model. Based on a subset dataset of the media data being outside a threshold accuracy, the subset dataset is automatically forwarded to a crowd source service. Verification of the subset dataset is received from the crowd source service, and the verified subset dataset is added to the first model. ## Claims ### Claim 1 A system for training a model comprising: at least one receiver for receiving media data; and at least one processor and at least one memory containing instructions that, when executed, cause the at least one processor to: separate the media data into one or more clusters, each cluster of the one or more clusters based on a feature from a first model; sample the media data of each cluster and, based on an analysis of the sampled media data, determine an accuracy of the media data of each cluster, the accuracy associated with the feature; based on a subset dataset of the media data being outside a threshold accuracy, automatically forward the subset dataset to a crowd source service; receive verification of the subset dataset from the crowd source service; and add the verified subset dataset to the first model. ### Claim 2 (depends on 1) The system of claim 1, the at least one processor further configured to: generate a second model based on the received verified subset dataset; and upon determining that an accuracy of the second model exceeds the first model, update the first model with the second model. ### Claim 3 (depends on 1) The system of claim 1, wherein the at least one processor that determines the accuracy of the subset dataset is configured to: determine a feature metric associated with the feature from an analysis of the first model; define a centroid based on the feature metric; and after separating the media data into the one or more clusters, determine a measured metric associated with the media data in each cluster. ### Claim 4 (depends on 3) The system of claim 3, wherein the at least one processor is further configured to: select the subset dataset of each cluster based on the measured metric matching the feature metric associated with the centroid within a threshold. ### Claim 5 (depends on 1) The system of claim 1, wherein the at least one processor is further configured to: receive a labelled subset dataset from the crowd source service; add the labelled subset dataset to the first model to create a combined model dataset; generate the second model based on the combined model dataset; and determine the accuracy of the second model. ### Claim 6 (depends on 1) The system of claim 1, wherein the at least one processor is configured to generate the second model on an ongoing basis. ### Claim 7 (depends on 1) The system of claim 1, wherein the at least one processor is configured to determine the feature from the first model based on at least one of an accent, gender, or environmental background noise in the media data. ### Claim 8 (depends on 1) The system of claim 1, wherein the media data is comprised of multiple data types, the multiple data types including audio, visual, and text data. ### Claim 9 (depends on 1) The system of claim 1, wherein the media data is configured to be separated into the one or more clusters based on an unsupervised machine learning technique. ### Claim 10 (depends on 1) The system of claim 1, wherein the at least one processor is further configured to automatically forward the subset dataset to the crowd source service based on a volume of the subset dataset being above the threshold accuracy. ### Claim 11 (depends on 10) The system of claim 10, wherein the volume of the subset dataset that initiates forwarding to the crowd source service is based on a volume heuristics model that is configured to determine an amount of data predicted to successfully update the first model. ### Claim 12 A non-transitory computer-readable medium comprising instructions, when executed by a computing system, the instructions cause the computing system to: separate media data into one or more clusters, each cluster of the one or more clusters based on a feature from a first model; sample the media data of each cluster and, based on an analysis of the sampled media data, determine an accuracy of the media data of each cluster, the accuracy associated with the feature; based on a subset dataset of the media data being outside a threshold accuracy, automatically forward the subset dataset to a crowd source service; receive verification of the subset dataset from the crowd source service; and add the verified subset dataset to the first model. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, the instructions further configured to: generate a second model based on the received verified subset dataset; and upon determining that an accuracy of the second model exceeds the first model, update the first model with the second model. ### Claim 14 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the instructions that determine the accuracy of the subset dataset are configured to: determine a feature metric associated with the feature from an analysis of the first model; define a centroid based on the feature metric; after separating the media data into the one or more clusters, determine a measured metric associated with the media data in each cluster; and select the subset dataset of each cluster based on the measured metric matching the feature metric associated with the centroid within a threshold. ### Claim 15 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein automatically forwarding the subset dataset to the crowd source service is further based on a volume of the subset dataset being above the threshold accuracy. ### Claim 16 A method of training a model comprising: separating media data into one or more clusters, each cluster of the one or more clusters based on a feature from a first model; sampling the media data of each cluster and, based on an analysis of the sampled media data, determining an accuracy of the mediadata of each cluster, the accuracy associated with the feature; based on a subset dataset of the media data being outside a threshold accuracy, automatically forwarding the subset dataset to a crowd source service; receiving verification of the subset dataset from the crowd source service; and adding the verified subset dataset to the first model. ### Claim 17 (depends on 16) The method of claim 16, further comprising: generating a second model based on the received verified subset dataset; and upon determining that an accuracy of the second model exceeds the first model, update the first model with the second model. ### Claim 18 (depends on 16) The method of claim 16, wherein determining the accuracy of the subset dataset comprises: determining a feature metric associated with the feature from an analysis of the first model; defining a centroid based on the feature metric; after separating the media data into the one or more clusters, determining a measured metric associated with the media data in each cluster; and selecting the subset dataset of each cluster based on the measured metric matching the feature metric associated with the centroid within a threshold. ### Claim 19 (depends on 16) The method of claim 16, wherein automatically forwarding the subset dataset to the crowd source service is further based on a volume of the subset dataset being above the threshold accuracy. ### Claim 20 (depends on 19) The method of claim 19, wherein the volume of the subset dataset that initiates forwarding to the crowd source service is based on a volume heuristics model that determines an amount of data predicted to successfully update the first model.