# US 2012/0254181 A1 · Text, character encoding and language recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0254181 A1
- Document ID: US20120254181A1
- Status: Published application
- Kind: A1
- Application number: 13/435,600
- Filed: 2012-03-30
- Published: 2012-10-04
- Earliest claimed: 2011-03-31
- Assignees: Clearswift Limited
- Inventors: Kevin Schofield; Istvan Biro
- Web page: https://patentgrep.com/patent/US20120254181A1

## CPC

- G06F 40/126: Character encoding
- G06F 40/263: Language identification
- G06Q 10/107: Computer-aided management of electronic mailing [e-mailing]
- H04L 51/212: using filtering or selective blocking
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A method is disclosed, for recognizing whether some electronic data is the digital representation of a piece of text and, if so, in which character encoding it has been encoded. A fingerprint is constructed from the data, wherein the fingerprint comprises, for each of a plurality of predetermined character encoding schemes, at least one confidence value, representing a confidence that the data was encoded using said character encoding scheme. The fingerprint also comprises a frequency value for each of a subset of byte values, each frequency value representing the frequency of occurrence of a respective byte value in the data. A statistical classification of the data is then performed based on the fingerprint.

## Claims

### Claim 1

A method for classifying data, the method comprising:
constructing a fingerprint from the data, wherein the fingerprint comprises:
for each of a plurality of predetermined character encoding schemes, at least one confidence value, representing a confidence that the data was encoded using said character encoding scheme; and
for each of a subset of byte values, a frequency value, each of said frequency value representing the frequency of occurrence of a respective byte value in the data, and
performing a statistical classification of the data based on the fingerprint.

### Claim 2 (depends on 1)

A method as claimed in claim 1, wherein the fingerprint comprises confidence values determined from examining bigrams in the data.

### Claim 3 (depends on 1)

A method as claimed in claim 1, wherein the fingerprint comprises confidence values determined from examining trigrams in the data.

### Claim 4 (depends on 1)

A method as claimed in claim 1, wherein the fingerprint comprises, for at least one of the plurality of predetermined character encoding schemes, a plurality of confidence values, each representing an independent assessment of confidence that the data was encoded using said encoding scheme.

### Claim 5 (depends on 4)

A method as claimed in claim 4, wherein the plurality of confidence values comprise a first confidence value determined from examining bigrams in the data and a second confidence value determined from examining trigrams in the data.

### Claim 6 (depends on 1)

A method as claimed in claim 1, comprising performing the statistical classification using a set of base classifiers whose results are aggregated using a meta-classifier or meta-algorithm such as Adaptive Boosting.

### Claim 7 (depends on 1)

A method as claimed in claim 1, wherein the step of performing the statistical classification comprises distinguishing textual data encoded in one of the predetermined character encoding schemes from non-textual data.

### Claim 8 (depends on 7)

A method as claimed in claim 7, further comprising, if it is determined that the data comprises textual data, identifying the character encoding scheme used for encoding said data.

### Claim 9 (depends on 8)

A method as claimed in claim 8, further comprising identifying the language represented by the textual data.

### Claim 10 (depends on 7)

A method as claimed in claim 7, further comprising, if it is determined that the data comprises non-textual data, identifying the type of non-textual data.

### Claim 11 (depends on 10)

A method as claimed in claim 10, further comprising identifying the type of non-textual data from a start sequence of the data.

### Claim 12 (depends on 1)

A method as claimed in claim 1, wherein said subset of byte values comprises byte values in the range A016FF16.

### Claim 13 (depends on 1)

A method of controlling data transfers, comprising:
classifying said data by means of a method according to claim 1; and
controlling the data transfer based on a result of the classification.

### Claim 14 (depends on 13)

A method as claimed in claim 13, comprising:
identifying textual data in said data;
identifying a language represented by the textual data; and
applying a language-specific policy to the data based on the identified language.

### Claim 15 (depends on 14)

A method as claimed in claim 14, wherein the step of applying a language-specific policy to the data comprises testing for the presence of certain words in a respective list for the identified language.

### Claim 16 (depends on 14)

A method as claimed in claim 14, wherein the data to be transferred comprises an email message, and wherein the step of applying a language-specific policy to the data comprises applying a language-specific test for spam.

### Claim 17 (depends on 13)

A method as claimed in claim 13, comprising identifying said data in a file.

### Claim 18 (depends on 13)

A method as claimed in claim 13, comprising identifying said data in a data stream.

### Claim 19

A computer program product, comprising computer readable code, suitable for causing a computer to perform a method for classifying data, the computer program product comprising:
first computer program code configured to construct a fingerprint from the data, wherein the fingerprint comprises:
for each of a plurality of predetermined character encoding schemes, at least one confidence value, representing a confidence that the data was encoded using said character encoding scheme; and
for each of a subset of byte values, a frequency value, each of said frequency value representing the frequency of occurrence of a respective byte value in the data, and
second computer program code configured to perform a statistical classification of the data based on the fingerprint.

### Claim 20 (depends on 19)

A computer system, comprising a computer program product as claimed in claim 19.
