# US 2013/0156302 A1 · Handwritten word spotter system using synthesized typed queries

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0156302 A1
- Document ID: US20130156302A1
- Status: Published application
- Kind: A1
- Application number: 13/678,961
- Filed: 2012-11-16
- Published: 2013-06-20
- Earliest claimed: 2008-09-18
- Assignees: Xerox Corporation
- Inventors: Jose A. Rodriguez Serrano; Florent C. Perronnin
- Web page: https://patentgrep.com/patent/US20130156302A1

## CPC

- G06V 30/2276: {with probabilistic networks, e.g. hidden Markov models}
- G06F 18/243: relating to the number of classes
- G06F 18/295: {Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models}
- G06V 30/19187: {Graphical models, e.g. Bayesian networks or Markov models}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A wordspotting system and method are disclosed for processing candidate word images extracted from handwritten documents. In response to a user inputting a selected query string, such as a word to be searched in one or more of the handwritten documents, the system automatically generates at least one computer-generated image based on the query string in a selected font or fonts. A model is trained on the computer-generated image(s) and is thereafter used in the scoring the candidate handwritten word images. The candidate or candidates with the highest scores and/or documents containing them can be presented to the user, tagged, or otherwise processed differently from other candidate word images/documents.

## Claims

### Claim 1

A method comprising:
receiving a query string;
generating a plurality of computer-generated images based on the query string with a computer typographic font, each of the plurality of computer-generated images being generated by varying the computer typographic font, the font variations having been identified based on a precision for retrieving word images which match at least one selected query string;
training a model based on the plurality of computer-generated images;
scoring candidate handwritten word images of a collection of handwritten word images using the trained model; and
based on the scores, identifying a subset of the word images.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
outputting the identified subset of word images.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
extracting the collection of candidate handwritten word images from at least one document.

### Claim 4 (depends on 3)

The method of claim 3, wherein the extracting comprises scanning at least one handwritten document and segmenting the scanned handwritten document.

### Claim 5 (depends on 1)

The method of claim 1, wherein the collection of candidate handwritten word images comprises images extracted from at least one document.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
implementing a computer implemented process for at least one document from which a word image of the identified subset of the candidate handwritten word images was extracted.

### Claim 7 (depends on 6)

The method of claim 6, wherein the computer implemented process comprises displaying at least that portion of the document from which the word image was extracted.

### Claim 8 (depends on 6)

The method of claim 6, wherein the computer implemented process comprises processing a document which contains one of the subset of candidate word images differently from a document which does not.

### Claim 9 (depends on 1)

The method of claim 1, wherein the receiving of the query string is performed prior to training of the model based on the plurality of computer-generated images.

### Claim 10 (depends on 1)

The method of claim 1, wherein the model comprises a first model which is trained on the plurality of computer-generated images and a second model which has been trained on a collection of sample handwritten word images and wherein the score of each word image is a function of a computed likelihood of the word image with respect to the first model and a computed likelihood of the word image with respect to the second model.

### Claim 11 (depends on 1)

The method of claim 1, wherein the model comprises a semi-continuous hidden Markov model and optionally further comprises a Gaussian mixture model.

### Claim 12 (depends on 1)

The method of claim 1, wherein the generating of the plurality of computer-generated images based on the query string comprises generating binary images of a fixed pixel height in pixels and wherein the candidate handwritten word images are normalized to the same fixed pixel height and are binary images.

### Claim 13 (depends on 1)

The method of claim 1, wherein the training of the model based on the plurality of computer-generated images comprises extracting features from patches of the plurality of computer-generated images.

### Claim 14 (depends on 13)

The method of claim 13, wherein the extracting features from patches of the plurality of computer-generated images comprises translating the image stepwise with a window of fixed height and width and at each step, extracting a patch.

### Claim 15 (depends on 1)

The method of claim 1, wherein:
the generating of the plurality of computer-generated images based on the query string includes, for each of a set of fonts, automatically generating images based on the query string;
the training of the model based on the plurality of computer-generated images includes modeling the query string with a semi-continuous hidden Markov model, a subset of the parameters of the semi-continuous hidden Markov model being estimated based on features extracted from the images in the different fonts, and other parameters of the semi-continuous hidden Markov model being previously trained on sample handwritten word images without consideration of the query string;
the scoring of the candidate handwritten word images includes scoring candidate handwritten word images of the collection against the trained semi-continuous hidden Markov model; and
the identifying of the subset of the word images based on the scores includes labeling one or more of the candidate handwritten word images, or a document containing one or more of the candidate handwritten word images.

### Claim 16 (depends on 15)

The method of claim 15, wherein the subset of parameters includes transition probabilities and mixture weights and the other parameters include means and covariance matrices.

### Claim 17 (depends on 1)

A non-transitory computer program product encoding instructions, which when executed on a computer causes the computer to perform the method of claim 1.

### Claim 18

A computer implemented processing system comprising:
a synthesizer which synthesizes a plurality of computer-generated images based on a received query string with a computer typographic font, each of the plurality of computer-generated images being generated by varying the computer typographic font, the font variations having been identified based on a precision for retrieving word images which match at least one selected query string;
a model which is trained on features extracted from the plurality of computer-generated images;
a scoring component which scores candidate handwritten word images of a collection of candidate handwritten word images against the model and, based on the scores, identifies a subset of the handwritten word images.

### Claim 19 (depends on 18)

The processing system of claim 18, further comprising:
a document segmentor which processes an input document image to identify the candidate handwritten word images.

### Claim 20 (depends on 18)

The processing system of claim 18, further comprising:
a features extractor which extracts the features of the plurality of computer-generated images and the candidate handwritten word images.

### Claim 21 (depends on 18)

The processing system of claim 18, further comprising:
a document tagger which labels documents or portions thereof which contain at least one of the subset of handwritten word images.

### Claim 22 (depends on 18)

The processing system of claim 18, wherein the model comprises a semi-continuous hidden Markov model which includes a background model trained on sample handwritten word images prior to receipt of a query string.

### Claim 23

A method comprising:
receiving a query string;
generating a plurality of computer-generated images based on the query string, each of the plurality of computer-generated images being generated by varying a selected computer typographic font;
training a model based on the plurality of computer-generated images in the different fonts;
scoring candidate handwritten word images of a collection of handwritten word images using the trained model; and
based on the scores, identifying a subset of the word images.

### Claim 24 (depends on 23)

The method of claim 23, further comprising identifying the set of fonts based on a precision of the set of fonts for retrieving word images which match at least one selected query string.