Back to results
US 10,706,322 B1Granted
Semantic ordering of image text
AssigneeAmazon Technologies, Inc.
InventorsShuo Yang, Hao Wu, Jonathan Wu, Meng Wang
Life of this patent
Estimating the term…
Abstract
Embodiments of the present disclosure provide systems and processes for automatically determining a layout of text within an image that makes sense from a semantic perspective. In certain embodiments, the systems disclosed herein receive bounding box information relating to one or more bounding boxes that surround text within the image. The systems compare the received bounding box information to determine a clustering of bounding boxes that have an above threshold probability of including words that when read in order make sense semantically. For example, systems herein can determine whether words in a cluster correspond to a line of text.
Claims20
A computer-implemented method comprising:
- as implemented by one or more computing devices executing specific computer-executable instructions comprising,
- identifying a set of words within an image, wherein the image comprises one or more words and one or more non-text objects;
- determining a bounding box for each word from the set of words to obtain a set of bounding boxes;
- determining an orientation vector for each bounding box of the set of bounding boxes, wherein the orientation vector of the bounding box corresponds to an orientation of the word located within the bounding box;
- for each pair of bounding boxes, determining an orientation difference between the bounding boxes of the pair of bounding boxes based at least in part on the orientation vector for each bounding box of the pair of bounding boxes;
- determining a minimum distance between bounding boxes for each pair of bounding boxes;
- determining a height of each bounding box;
- determining a center distance for each pair of bounding boxes, wherein the center distance comprises a distance between a center of each bounding box of the pair of bounding boxes;
- performing an iterative clustering process based on the orientation of each bounding box, the orientation difference for each pair of bounding boxes, the minimum distance between bounding boxes for each pair of bounding boxes, the height of each bounding box, and the center distance for each pair of bounding boxes to obtain a plurality of clusters of words from the set of words, wherein each cluster corresponds to a different line of words presented in the image;
- selecting a cluster of words from the plurality of clusters based on cluster selection criteria, the cluster selection criteria including a size of the cluster and a location of the cluster in the image;
- obtaining a rotated cluster by performing orthogonal projection on the cluster of words selected from the plurality of clusters to rotate the cluster of words along a horizontal axis and align the cluster of words with the horizontal axis;
- performing a raster scanning operation on the rotated cluster to determine the words included in the rotated cluster;
- and
- outputting the words to a selected target application that performs application-specific text processing.
11 more elements
A system comprising:
- an electronic data store to at least store an image comprising text and non-text graphics;
- a computing system comprising computer hardware in communication with the electronic data store, the computing system to execute specific computer-executable instructions to at least:
- receive a set of coordinates for each of a set of bounding boxes in the image;
- derive a set of comparative data for the set of bounding boxes based at least in part on the set of coordinates for each bounding box, the set of comparative data derived by comparing a bounding box from the set of bounding boxes to at least one other bounding box from the set of bounding boxes, wherein the set of comparative data includes an orientation difference between a pair of bounding boxes obtained by comparing orientation vectors derived for each bounding box of the pair of bounding boxes;
- divide the set of bounding boxes into a number of bounding box clusters based at least in part on the set of comparative data;
- select a bounding box cluster based on a set of selection criteria comprising a size of the bounding box cluster and a location of the bounding box cluster in the image;
- perform a text recognition operation on the bounding box cluster to recognize text included in the bounding box cluster;
- and
- output the text to a target application.
6 more elements
One or more non-transitory, computer-readable storage media storing computer-executable instructions, which if performed by one or more hardware processors, cause the one or more hardware processors to at least:
- receive four sets of coordinate pairs for each of a set of bounding boxes in an image that depicts text and non-text objects, the four sets of coordinate pairs corresponding to four corners of a bounding box;
- derive a set of metadata for the set of bounding boxes based at least in part on the four sets of coordinate pairs for each bounding box, at least some of the set of metadata determined by comparing the four sets of coordinate pairs of a first bounding box to the four sets of coordinate pairs of a second bounding box from the set of bounding boxes wherein the set of metadata includes an orientation difference between a pair of bounding boxes obtained by comparing orientation vectors derived for each bounding box of the pair of bounding boxes;
- cluster the set of bounding boxes into a number of bounding box clusters based at least in part on the set of metadata;
- select a bounding box cluster based at least in part on an expected amount of processor utilization to process text included in bounding boxes of the bounding box cluster;
- perform a text recognition operation on the bounding box cluster to recognize text included in the bounding box cluster;
- and
- cause the text to be output for subsequent processing.
4 more elements
Dependencies are inferred from the claim text.