# US 2018/0024974 A1 · Enhancing documents portrayed in digital images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0024974 A1
- Document ID: US20180024974A1
- Status: Published application
- Kind: A1
- Application number: 15/658,289
- Filed: 2017-07-24
- Published: 2018-01-25
- Earliest claimed: 2016-07-22
- Assignees: Dropbox, Inc.
- Applicants: Dropbox, Inc.
- Inventors: Nils Peter Welinder; Peter N. Belhumeur; Ying Xiong; Jongmin Baek; Simon Kozlov; Thomas Berg; David J. Kriegman
- Web page: https://patentgrep.com/patent/US20180024974A1

## CPC

- G06F 40/123: Storage facilities
- G06F 16/93: Document management systems
- G06F 40/106: Display of layout of documents; Previewing
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/197: Version control (for software G06F8/71)
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/90: Dynamic range modification of images or parts thereof
- G06T 7/194: involving foreground-background segmentation
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06V 10/10: Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60)
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 30/40: Document-oriented image-based pattern recognition
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/126: {Job scheduling, e.g. queuing, determine appropriate device}
- G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191)
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06T 5/70: Denoising; Smoothing
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30176: Document
- G06T 2210/22: Cropping
- G06V 10/242: {by image rotation, e.g. by 90 degrees}
- G06V 10/30: Noise filtering
- G06V 30/418: Document matching, e.g. of document images
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure is directed toward systems and methods that efficiently and effectively generate an enhanced document image of a displayed document in an image frame captured from a live image feed. For example, systems and methods described herein apply a document enhancement process to a displayed document in an image frame that result in an enhanced document image that is cropped, rectified, un-shadowed, and with dark text against a mostly white background. Additionally, systems and method described herein determine whether a stored digital content item includes a displayed document. In response to determining that a stored digital content item does include a displayed document, systems and methods described herein generate an enhanced document image of a displayed document included in the stored digital content item.

## Claims

### Claim 1

A computing device comprising:
at least one processor; and
a non-transitory computer-readable medium storing instructions thereon that, when executed by the at least one processor, cause the computing device to:
detect a displayed document within a live image feed associated with the computing device;
based on detecting the displayed document within the live image feed, generate an enhanced document image corresponding to the displayed document; and
provide, for presentation on a display of the computing device, the enhanced document image.

### Claim 2 (depends on 1)

The computing device as recited in claim 1, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to:
capture an image frame from the live image feed, the image frame comprising the displayed document; and
wherein generating the enhanced document comprises modifying the image frame with respect to the displayed document within the image frame.

### Claim 3 (depends on 2)

The computing device as recited in claim 2, wherein modifying the image frame comprises:
detecting, without receiving user input, portions of the image frame that are not part of the displayed document; and
cropping the image frame to remove the portions of the image frame that are not part of the displayed document.

### Claim 4 (depends on 3)

The computing device as recited in claim 3, wherein generating the enhanced document further comprises altering the displayed document within the cropped imaged frame.

### Claim 5 (depends on 4)

The computing device of claim 4, wherein altering the displayed document comprises at least one of: rectifying the displayed document, converting the displayed document to grayscale, or denoising the displayed document.

### Claim 6 (depends on 4)

The computing device as recited in claim 4, wherein altering the displayed document comprises correcting a background of the displayed document.

### Claim 7 (depends on 6)

The computing device as recited in claim 6, wherein correcting the background of the displayed document comprises:
creating a subsampled version of the displayed document; and
optimizing the subsampled version of the displayed document by solving an objective function that penalizes deviations from white within the subsampled version and penalizes deviations in gradient within the subsampled version to generate an optimized subsampled version.

### Claim 8 (depends on 7)

The computing device as recited in claim 7, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to:
perform a Fourier Domain transfer of the subsampled version of the displayed document;
solve the objective function in the Fourier Domain; and
perform an inverse Fourier Domain transfer to generate the optimized subsampled version of the displayed document.

### Claim 9 (depends on 7)

The computing device as recited in claim 7, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to upsample the optimized subsampled version of the displayed document to generate a tri-map version of the displayed document that identifies background pixels, foreground pixels, and unknown pixels.

### Claim 10 (depends on 9)

The computing device as recited in claim 9, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to assign each of the unknown pixels as either a background pixel or a foreground pixel by estimating a background color of each of the unknown pixels.

### Claim 11 (depends on 10)

The computing device as recited in claim 10, wherein the non-transitory computer-readable medium further comprises instructions thereon that, when executed by the at least one processor, cause the computing device to convert the enhanced document image to a document file format.

### Claim 12

A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:
detect a displayed document within a live image feed associated with the computing device;
based on detecting the displayed document within the live image feed, generate an enhanced document image corresponding to the displayed document; and
provide, for presentation on a display of the computer system, the enhanced document image.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium recited in claim 12, further comprising instructions that, when executed by the at least one processor, cause the computer system to:
receive user input indicating one or more edits to the enhanced document image;
modify the enhanced document image in accordance with the one or more edits; and
convert the enhanced document image to a document file format.

### Claim 14 (depends on 12)

The non-transitory computer-readable medium recited in claim 12, wherein generating the enhanced document image comprises altering a border of the displayed document to create a rectangular enhanced document image.

### Claim 15 (depends on 12)

The non-transitory computer-readable medium recited in claim 12, wherein generating the enhanced document comprises:
converting the displayed document from a color version to a grayscale version; and
recoloring the displayed document prior to providing the enhanced document image.

### Claim 16

A method comprising:
receiving, at an online content management system and from a client device, a digital content item;
determining, by at least one processor, that the digital content item comprises a displayed document;
associating metadata with the digital content item that designates the digital content item as comprising the displayed document;
based on the metadata associated with the digital content item, providing a selectable graphical element associated with the digital content item within a graphical user interface on the client device, wherein the selectable graphical element indicates that the digital content item comprises the displayed document; and
in response to receiving an indication of a user selection of the selectable graphical element, generating an enhanced document image for the displayed document.

### Claim 17 (depends on 16)

The method as recited in claim 16, wherein generating an enhanced document image for the displayed document within the digital content item comprises one or more of:
cropping the digital content item with respect to the displayed document; rectifying the displayed document; converting the displayed document to grayscale; or denoising the displayed document.

### Claim 18 (depends on 16)

The method as recited in claim 16, further comprising:
providing the enhanced document image to the client device; and
storing the enhanced document image on the online content management system in an account associated with the client device.

### Claim 19 (depends on 18)

The method as recited in claim 18, further comprising:
training an image classifier; and
wherein determining that the digital content item comprises the displayed document comprises analyzing the digital content item using the trained image classifier.

### Claim 20 (depends on 19)

The method as recited in claim 19, wherein building and training the image classifier comprises:
building a convolutional neural network comprising a plurality of layers; and
training, utilizing a training data set, the convolutional neural network to determine whether an image includes a displayed document.