# US 2017/0364746 A1 · Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0364746 A1
- Document ID: US20170364746A1
- Status: Published application
- Kind: A1
- Application number: 15/679,237
- Filed: 2017-08-17
- Published: 2017-12-21
- Earliest claimed: 2004-09-27
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Martin T. King; Dale L. Grover; Clifford A. Kushler; James Q. Stafford-Fraser
- Web page: https://patentgrep.com/patent/US20170364746A1

## CPC

- H04N 1/32101: {Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title}
- G06F 16/5846: {using extracted text}
- G06Q 10/10: Office automation; Time management
- G06Q 30/00: Commerce
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 20/66: Trinkets, e.g. shirt buttons or jewellery items (recognising microscopic objects G06V20/69)
- G06V 30/142: using hand-held instruments; Constructional details of the instruments
- G06V 30/36: {Matching; Classification}
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- G06V 30/418: Document matching, e.g. of document images
- H04N 1/107: with manual scanning
- H04N 1/00241: {using an image reading device as a local input to a computer (input via network, e.g. network scanners H04N1/00225)}
- H04N 1/00307: {with a mobile telephone apparatus (constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00)}
- H04N 1/00488: {providing an audible output to the user}
- H04N 1/0049: {providing a visual indication to the user, e.g. using a lamp (H04N1/00408 takes precedence)}
- H04N 2201/0041: Point to point (H04N2201/0039 takes precedence)
- H04N 2201/0055: By radio
- H04N 2201/0058: Docking-station, cradle or the like
- H04N 2201/0081: Image reader (H04N2201/0091 - H04N2201/0094 take precedence)
- H04N 2201/0084: Digital still camera
- H04N 2201/0096: Portable devices
- H04N 2201/3249: data relating to a linked page or object, e.g. hyperlink
- H04N 2201/3264: of sound signals
- H04N 2201/3269: of machine readable codes or marks, e.g. bar codes or glyphs
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A device for capturing rendered text is described. The device incorporates one or more visual sensors that receive visual information as a part of capturing rendered text. The visual sensors are collectively capable of capturing both text that is permanently printed on a page, and text that is displayed transitorily on a dynamic device. The device further incorporates a visual information disposition subsystem for disposing of visual information received by the visual sensors. The device further incorporates a package that bears the visual sensors and the visual information disposition subsystem, and is suitable to be held in a human hand.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A computer-implemented method comprising:
receiving, by a server system and from a mobile computing device, an image of at least a portion of a rendered document that was captured by an optical capture component of the mobile computing device, wherein the image depicts content of the rendered document;
receiving, by the server system and from the mobile computing device, an annotation for the image, the annotation having been captured by an audio capture component of the mobile computing device;
identifying, by the server system, based on the content depicted in the image of the at least a portion of the rendered document, a digital document that corresponds to the rendered document; and
modifying, by the server system, the identified digital document such that the digital document includes annotation data that is associated with the received annotation for the image.

### Claim 22 (depends on 21)

The method of claim 21, wherein identifying the digital document comprises:
(i) performing optical recognition on the image to recognize text included in the at least a portion of the rendered document; and
(ii) comparing the optically recognized text with contents of a corpus of electronic documents to identify a particular digital document in the corpus that corresponds to the rendered document.

### Claim 23 (depends on 21)

The method of claim 21, further comprising:
determining a point in the digital document that corresponds to the content of the rendered document depicted in the image.

### Claim 24 (depends on 23)

The method of claim 23, wherein modifying the identified digital document such that the digital document includes annotation data comprises:
attaching an audio clip included in the received annotation to the determined point in the digital document.

### Claim 25 (depends on 23)

The method of claim 23, wherein modifying the identified digital document such that the digital document includes annotation data comprises:
(i) performing voice recognition on the received annotation to generate a voice-recognized text version of the annotation; and
(ii) attaching the voice-recognized text version of the annotation to the determined point in the digital document.

### Claim 26 (depends on 23)

The method of claim 23, wherein modifying the identified digital document such that the digital document includes annotation data comprises:
(i) performing voice recognition on the received annotation to generate a voice-recognized text version of the annotation; and
(ii) performing one or more edits to the determined point in the digital document based on content included in the voice-recognized text version of the annotation.

### Claim 27 (depends on 21)

The method of claim 21, further comprising:
determining that a user of the mobile computing device has sufficient privileges to modify the identified digital document, wherein modifying the identified digital document is performed in response to the determining.

### Claim 28 (depends on 21)

The method of claim 21, further comprising:
storing, by the server system, a record of the received annotation for the image, wherein the record includes the image or an optically recognized text version of the image, and includes the received annotation or a voice-recognized text version of the annotation.

### Claim 29 (depends on 21)

The method of claim 21, further comprising:
receiving, by the server system and from a different mobile computing device, a request for the digital document; and
providing, by the server system and to the different mobile computing device, the digital document including the annotation data.

### Claim 30

Non-transitory computer-readable media having contents capable of causing a computing system including one or more computers to perform operations comprising:
receiving, by a server system and from a mobile computing device, an image of at least a portion of a rendered document that was captured by an optical capture component of the mobile computing device, wherein the image depicts content of the rendered document;
receiving, by the server system and from the mobile computing device, an annotation for the image, the annotation having been captured by an audio capture component of the mobile computing device;
identifying, by the server system, based on the content depicted in the image of the at least a portion of the rendered document, a digital document that corresponds to the rendered document; and
modifying, by the server system, the identified digital document such that the digital document includes annotation data that is associated with the received annotation for the image.

### Claim 31 (depends on 30)

The non-transitory computer-readable media of claim 30, the operations further comprising:
determining a point in the digital document that corresponds to the content of the rendered document depicted in the image.

### Claim 32 (depends on 31)

The non-transitory computer-readable media of claim 31, wherein modifying the identified digital document such that the digital document includes annotation data comprises:
attaching an audio clip included in the received annotation to the determined point in the digital document.

### Claim 33 (depends on 31)

The non-transitory computer-readable media of claim 31, wherein modifying the identified digital document such that the digital document includes annotation data comprises:
(i) performing voice recognition on the received annotation to generate a voice-recognized text version of the annotation; and
(ii) attaching the voice-recognized text version of the annotation to the determined point in the digital document.

### Claim 34 (depends on 31)

The non-transitory computer-readable media of claim 31, wherein modifying the identified digital document such that the digital document includes annotation data comprises:
(i) performing voice recognition on the received annotation to generate a voice-recognized text version of the annotation; and
(ii) performing one or more edits to the determined point in the digital document based on content included in the voice-recognized text version of the annotation.

### Claim 35

A system, comprising one or more computers having one or more memory devices on which are stored computer executable instructions operable to cause the one or more computers to perform operations comprising:
receiving, by a server system and from a mobile computing device, an image of at least a portion of a rendered document that was captured by an optical capture component of the mobile computing device, wherein the image depicts content of the rendered document;
receiving, by the server system and from the mobile computing device, an annotation for the image, the annotation having been captured by an audio capture component of the mobile computing device;
identifying, by the server system, based on the content depicted in the image of the at least a portion of the rendered document, a digital document that corresponds to the rendered document; and
modifying, by the server system, the identified digital document such that the digital document includes annotation data that is associated with the received annotation for the image.

### Claim 36 (depends on 35)

The system of claim 35, the operations further comprising:
determining a point in the digital document that corresponds to the content of the rendered document depicted in the image.

### Claim 37 (depends on 36)

The system of claim 36, wherein modifying the identified digital document such that the digital document includes annotation data comprises:
attaching an audio clip included in the received annotation to the determined point in the digital document.

### Claim 38 (depends on 36)

The system of claim 36, wherein modifying the identified digital document such that the digital document includes annotation data comprises:
(i) performing voice recognition on the received annotation to generate a voice-recognized text version of the annotation; and
(ii) attaching the voice-recognized text version of the annotation to the determined point in the digital document.

### Claim 39 (depends on 36)

The system of claim 36, wherein modifying the identified digital document such that the digital document includes annotation data comprises:
(i) performing voice recognition on the received annotation to generate a voice-recognized text version of the annotation; and
(ii) performing one or more edits to the determined point in the digital document based on content included in the voice-recognized text version of the annotation.
