System and method for real-time data categorization
AssigneeIncucomm, Inc.
InventorChristopher Scott Heinlen
Life of this patent
Estimating the term…
Abstract
A data categorization system and corresponding method for real-time data categorization of streaming data. In one embodiment, the system checking the streaming data against known data clusters and assign ones of the streaming data that fit the known data clusters thereto, otherwise categorize the ones of the streaming data as unclassified data. The system executes unsupervised clustering on the unclassified data to generate new data clusters. The system defines shells for the new data clusters to add to the known data clusters. The data system outputs the new data clusters to a data analysis system to execute a model to operate a real system with computational efficiency in real-time.
Claims20
A data categorization system for dynamically categorizing streaming data into data clusters, said system having no prior knowledge of said data clusters to which said streaming data can be assigned, said data categorization system being operable on a processor and memory configured to:
- check said streaming data against known data clusters and assign ones of said streaming data that fit said known data clusters thereto, otherwise categorize said ones of said streaming data as unclassified data;
- execute unsupervised clustering on said unclassified data when said unclassified data reaches a threshold, said unsupervised clustering configured to:
- generate a minimum spanning tree of said unclassified data,
- generate a dendrogram from said minimum spanning tree including nodes having ones of said unclassified data,
- determine which nodes are viable as new data clusters, and
- cull noise and extraneous ones of said unclassified data from said new data clusters, said unsupervised clustering producing a transformed and reduced set of said unclassified data within said new data clusters;
- define shells for said new data clusters, configured to:
- determine a transformation that forces said unclassified data into a unit sphere or cube,
- apply said transformation to each of said unclassified data, and
- assign any of said unclassified data that fall within said unit sphere or cube to said shells for said new data clusters;
- said shells including a transformed and reduced set of said unclassified data to add to said known data clusters;
- and
- output said new data clusters to a data analysis system to execute a model to operate a real system with computational efficiency in real-time.
A method of operating a data categorization system for dynamically categorizing streaming data into data clusters, said data categorization system having no prior knowledge of said data clusters to which said streaming data can be assigned, said method, comprising:
- checking said streaming data against known data clusters and assign ones of said streaming data that fit said known data clusters thereto, otherwise categorize said ones of said streaming data as unclassified data;
- executing unsupervised clustering on said unclassified data when said unclassified data reaches a threshold, comprising:
- generating a minimum spanning tree of said unclassified data,
- generating a dendrogram from said minimum spanning tree including nodes having ones of said unclassified data,
- determining which nodes are viable as new data clusters, and
- culling noise and extraneous ones of said unclassified data from said new data clusters,
- said unsupervised clustering producing a transformed and reduced set of said unclassified data within said new data clusters;
- defining shells for said new data clusters, comprising:
- determining a transformation that forces said unclassified data into a unit sphere or cube,
- applying said transformation to each of said unclassified data, and
- assigning any of said unclassified data that fall within said unit sphere or cube to said shells for said new data clusters;
- said shells including a transformed and reduced set of said unclassified data to add to said known data clusters;
- and
- outputting said new data clusters to a data analysis system to execute a model to operate a real system with computational efficiency in real-time.
Dependencies are inferred from the claim text.