# US 2014/0195497 A1 · Real-time identification of data candidates for classification based compression > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0195497 A1 - Document ID: US20140195497A1 - Status: Published application - Kind: A1 - Application number: 13/738,262 - Filed: 2013-01-10 - Published: 2014-07-10 - Earliest claimed: 2013-01-10 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Jonathan AMIT; Lilia DEMIDOV; George GOLDBERG; Nir HALOWANI; Ronen I. KAT; Chaim KOIFMAN; Sergey MARENKOV; Dmitry SOTNIKOV - Web page: https://patentgrep.com/patent/US20140195497A1 ## CPC - G06F 16/1744: {using compression, e.g. sparse files} - G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors - G06F 16/2365: {Ensuring data consistency and integrity} - G06F 16/285: {Clustering or classification} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate for determining if the data candidate may benefit from the classification-based compression. A decision is provided for approving the classification-based compression on the data candidates according to the heuristic. ## Claims ### Claim at position 1 1-8. (canceled) ### Claim 9 A system for real-time identification of data candidates for data processing in a computing environment, the system comprising: a processor device operable in the computing storage environment, wherein the processor device: samples data candidates for performing a classification-based compression upon the data candidates, and computes a heuristic on a randomly selected data sample from the data candidate thereby determining if the data candidates may benefit from the classification-based compression, wherein a decision is provided for approving the classification-based compression on the data candidates according to the heuristic. ### Claim 10 (depends on 9) The system of claim 9, wherein the processor device estimates if data classes are present in the data candidates. ### Claim 11 (depends on 10) The system of claim 10, wherein the processor device, if data classes are known in advance, computes the heuristic by: for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, wherein the characters that do not belong to the predefined set of the data classes are non-classifiable data, and calculating an actual number of the characters for each of the data classes and the non-classifiable data, wherein a minimum, a maximum, an average, and a distribution of a number of consecutive characters in each of the data classes are calculated. ### Claim 12 (depends on 11) The system of claim 11, wherein the processor device: sums a ratio between the actual number of the characters and the expected number of the characters, and then dividing the ratio by a number of the data classes that are not empty, wherein the non-classifiable data are included in the number of the data classes during the dividing, and the number of the data classes, that are not empty, have characters that belong to the class that were observed in the input, and performs the classification-based compression on the data candidates if the ratio exceeds a threshold. ### Claim 13 (depends on 10) The system of claim 10, wherein the processor device, if data classes are not known in advance: calculates a pair-wise histogram by counting a number of appearances for each two-consecutive characters for creating a graph representation having weighted edges, wherein a weight is the number of appearances of the two-consecutive characters, runs a clustering operation on the graph representation whereby the clustering operation filters outliers, and each cluster in the graph representation is a data class, and executes the heuristic computed for data classes that are known in advance for performing the classification-based compression if more than one cluster is identified, wherein the classification-based compression is not performed if the clustering operation outputs a single cluster. ### Claim 14 (depends on 9) The system of claim 9, wherein the processor device: analyzes data in a write buffer independently for specific character types, and performs the classification-based compression if a ratio of the specific character types exceeds non-specific character types by a threshold amount. ### Claim 15 (depends on 9) The system of claim 9, wherein the processor device performs a classification identification on the data candidates during at least one single pass for both fixed data lengths and variable data lengths. ### Claim 16 (depends on 9) The system of claim 9, wherein the data candidates are data blocks for determining intra-block classification of the data for performing the classification-based compression on the data candidates. ### Claim 17 A computer program product for real-time identification of data candidates for data processing by a processor device, the computer program product comprising a computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising: a first executable portion that samples data candidates for performing a classification-based compression upon the data candidates, and a second executable portion that computes a heuristic on a randomly selected data sample from the data candidate thereby determining if the data candidates may benefit from the classification-based compression, wherein a decision is provided for approving the classification-based compression on the data candidates according to the heuristic. ### Claim 18 (depends on 17) The computer program product of claim 17, further including a third executable portion that estimates if data classes are present in the data candidates. ### Claim 19 (depends on 18) The computer program product of claim 18, further including a fourth executable portion that, if data classes are known in advance, computes the heuristic by: for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, wherein the characters that do not belong to the predefined set of the data classes are non-classifiable data, and calculating an actual number of the characters for each of the data classes and the non-classifiable data, wherein a minimum, a maximum, an average, and a distribution of a number of consecutive characters in each of the data classes are calculated. ### Claim 20 (depends on 19) The computer program product of claim 19, further including a fifth executable portion that: sums a ratio between the actual number of the characters and the expected number of the characters, and then dividing the ratio by a number of the data classes that are not empty, wherein the non-classifiable data are included in the number of the data classes during the dividing, and the number of the data classes, that are not empty, have characters that belong to the class that were observed in the input, and performs the classification-based compression on the data candidates if the ratio exceeds a threshold. ### Claim 21 (depends on 18) The computer program product of claim 18, further including a fourth executable portion that, if data classes are not known in advance: calculates a pair-wise histogram by counting a number of appearances for each two-consecutive characters for creating a graph representation having weighted edges, wherein a weight is the number of appearances of the two-consecutive characters, runs a clustering operation on the graph representation whereby the clustering operation filters outliers, and each cluster in the graph representation is a data class, and executes the heuristic computed for data classes that are known in advance for performing the classification-based compression if more than one cluster is identified, wherein the classification-based compression is not performed if the clustering operation outputs a single cluster. ### Claim 22 (depends on 17) The computer program product of claim 17, further including a third executable portion that: analyzes data in a write buffer independently for specific character types, and performs the classification-based compression if a ratio of the specific character types exceeds non-specific character types by a threshold amount. ### Claim 23 (depends on 17) The computer program product of claim 17, further including a third executable portion that performs a classification identification on the data candidates during at least one single pass for both fixed data lengths and variable data lengths. ### Claim 24 (depends on 17) The computer program product of claim 17, wherein the data candidates are data blocks for determining intra-block classification of the data for performing the classification-based compression on the data candidates.