# US 10,872,271 B2 · Training image-processing neural networks by synthetic photorealistic indicia-bearing images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,271 B2
- Document ID: US10872271B2
- Status: Granted
- Kind: B2
- Application number: 16/137,981
- Filed: 2018-09-21
- Published: 2020-03-19
- Granted: 2020-12-22
- Earliest claimed: 2018-09-19
- Assignees: Abbyy Production LLC
- Applicants: ABBYY Production LLC
- Inventors: Ivan Germanovich Zagaynov; Pavel Valeryevich Borin
- Web page: https://patentgrep.com/patent/US10872271B2

## CPC

- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/2431: Multiple classes
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 5/30: Erosion or dilatation, e.g. thinning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 5/90: Dynamic range modification of images or parts thereof
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/764: using classification, e.g. of video objects
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 30/19173: {Classification techniques}
- G06T 5/75: Unsharp masking
- G06T 2207/20024: Filtering details
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20212: Image combination
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for training image processing neural networks by synthetic photorealistic indicia-bearing images. An example method comprises: generating an initial set of images, wherein each image of the initial set of images comprises a rendering of a text string; producing an augmented set of images by processing the initial set of images to introduce, into each image of the initial set of image, at least one simulated image defect; generating a training dataset comprising a plurality of pairs of images, wherein each pair of images comprises a first image selected from the initial set of images and a second image selected from the augmented set of images; and training, using the training dataset, a convolutional neural network for image processing.

## Claims

### Claim 1

A method, comprising:
generating, by a computer system, an initial set of images, wherein each image of the initial set of images comprises a rendering of a text string;
producing an augmented set of images by processing the initial set of images to introduce, into each image of the initial set of images, at least one simulated image defect, by applying, to at least a subset of pixels of the image, a simulated digital noise;
generating a training dataset comprising a plurality of pairs of images, wherein each pair of images comprises a first image selected from the initial set of images and a second image selected from the augmented set of images; and
training, using the training dataset, a convolutional neural network for image processing.

### Claim 2 (depends on 1)

The method of claim 1, wherein processing the initial set of images further comprises:
superimposing, on a generated image, a transparent image of a pre-defined or randomly generated text.

### Claim 3 (depends on 1)

The method of claim 1, wherein processing the initial set of images further comprises:
de-contrasting a generated image to reduce a maximum difference in luminance of pixels of the generated image by a pre-defined value.

### Claim 4 (depends on 1)

The method of claim 1, wherein processing the initial set of images further comprises:
simulating an additional light source in a scene of a generated image by additively applying, to at least a subset of pixels of the generated image, low frequency Gaussian noise of a low amplitude.

### Claim 5 (depends on 1)

The method of claim 1, wherein processing the initial set of images further comprises:
de-focusing a generated image by applying, to at least a subset of pixels of a generated image, Gaussian blur.

### Claim 6 (depends on 1)

The method of claim 1, wherein processing the initial set of images further comprises:
simulating movement of imaged objects in a generated image by superimposing a motion blur on the generated image.

### Claim 7 (depends on 1)

The method of claim 1, wherein processing the initial set of images further comprises:
simulating camera pre-processing of a generated image by applying a filter to at least a subset of pixels of the generated image.

### Claim 8 (depends on 1)

The method of claim 1, wherein processing the initial set of images further comprises:
simulating de-mosaicing of a generated image by applying Gaussian blur to at least a subset of pixels of the generated image.

### Claim 9 (depends on 1)

The method of claim 1, wherein the convolutional neural network comprises multiple convolution layers which implement dilated convolution operators with various dilation parameter values.

### Claim 10 (depends on 9)

The method of claim 9, wherein training the convolutional neural network further comprises:
determining a per-parameter training rate by calculating an exponential moving average of a gradient and a squared gradient of an input signal.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
utilizing the convolution neural network for image pre-processing for an optical character recognition (OCR) application.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
generating a classification convolutional neural network for classifying a set of input images into a first class comprising synthetic images and a second class comprising real photo images.

### Claim 13 (depends on 12)

The method of claim 12, wherein generating a classification convolutional neural network comprises modifying a convolution neural network utilized for image pre-processing.

### Claim 14 (depends on 12)

The method of claim 12, further comprising:
utilizing the classification convolutional neural network for filtering the training data set.

### Claim 15

A system, comprising:
a memory;
a processor, coupled to the memory, the processor configured to:
generate an initial set of images, wherein each image of the initial set of images comprises a rendering of a text string;
produce an augmented set of images by processing the initial set of images to introduce, into each image of the initial set of images, at least one simulated image defect;
generate a training dataset comprising a plurality of pairs of images, wherein each pair of images comprises a first image selected from the initial set of images and a second image selected from the augmented set of images; and
train, using the training dataset, a convolutional neural network for image processing, wherein the convolutional neural network comprises a preprocessing branch including a set of convolution filters performing local transformations of an input image and a context branch including multiple convolution layers which reduce the input image by a scaling factor and multiple trans-convolution layers for enlarging the image by the scaling factor.

### Claim 16

A computer-readable non-transitory storage medium comprising executable instructions that, when executed by a processing device, cause the processing device to:
generate an initial set of images, wherein each image of the initial set of images comprises a rendering of a text string;
produce an augmented set of images by processing the initial set of images to introduce, into each image of the initial set of image, at least one simulated image defect;
generate a training dataset comprising a plurality of pairs of images, wherein each pair of images comprises a first image selected from the initial set of images and a second image selected from the augmented set of images; and
train, using the training dataset, a convolutional neural network for image processing, wherein training the convolutional neural network is performed using a hinge loss function.

### Claim 17 (depends on 16)

The computer-readable non-transitory storage medium of claim 16, further comprising executable instructions causing the processing device to:
generate a classification convolutional neural network for classifying a set of input images into a first class comprising synthetic images and a second class comprising real photo images.