# US 2010/0166309 A1 · System And Methods For Creation And Use Of A Mixed Media Environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0166309 A1
- Document ID: US20100166309A1
- Status: Published application
- Kind: A1
- Application number: 12/719,437
- Filed: 2010-03-08
- Published: 2010-07-01
- Earliest claimed: 2004-10-01
- Assignees: Ricoh Company, Ltd.
- Inventors: Jonathan J. Hull; Berna Erol; Jamey Graham; Peter E. Hart; Dar-Shyang Lee; Kurt Piersol
- Web page: https://patentgrep.com/patent/US20100166309A1

## CPC

- G06F 16/583: using metadata automatically derived from the content
- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- G06F 21/78: to assure secure storage of data (address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00)
- G06F 21/80: in storage media based on magnetic or optical technology, e.g. disks with sectors (preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00)
- G06F 40/12: Use of codes for handling textual entities
- G06F 40/123: Storage facilities
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 30/142: using hand-held instruments; Constructional details of the instruments
- G06V 30/268: {Lexical context}
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/418: Document matching, e.g. of document images
- G06V 30/10: Character recognition
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system includes an MMR user, a MMR computer, a user printer that produces a printed document, a networked media server, an office portal, a service provider server, an electronic display that is electrically connected to a set-top box, a document scanner, a network, a capture device, a cellular infrastructure, wireless fidelity (Wi-Fi) technology, Bluetooth® technology, infrared (IR) technology, wired technology, and a geo location mechanism. The MMR system of the present invention provides mechanisms for forming a mixed media document that includes media of at least two types, such as printed paper as a first medium and a digital photograph, digital movie, digital audio file, or web link as a second medium. Furthermore, the MMR system of the present invention facilitates business methods that take advantage of the combination of a portable electronic device, such as a cellular camera phone, and a paper document.

## Claims

### Claim 1

A system for creating a mixed media reality document data structure, the system comprising:
a capture device that captures an image patch of a printed document;
a mixed media processor communicatively coupled to the capture device to receive the image patch and to compare the captured image patch to a previously generated mixed media reality document data structure to relate the printed document with the mixed media reality document data structure, the mixed media reality document data structure comprising:
a representation of the printed document comprising a matchable portion for use in pattern matching to a corresponding first location within the captured image patch;
an electronic representation of the printed document; and
an index correlating the matchable portion of the representation with a second location in the electronic document; and
a media storage communicatively coupled to the mixed media processor that stores the mixed media reality document data structure.

### Claim 2 (depends on 1)

The system of claim 1, wherein the mixed media processor is adapted to relate the image patch to the mixed media reality document data structure and perform an action in response to a relationship between the image patch and the mixed media reality document data structure.

### Claim 3 (depends on 2)

The system of claim 2, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

### Claim 4 (depends on 1)

The system of claim 1, wherein the media storage stores a plurality of mixed media reality document data structures each having a representation of a printed document and an index, the media storage coupled to the mixed media processor.

### Claim 5 (depends on 1)

The system of claim 1, wherein the image patch is a portion of the printed document.

### Claim 6 (depends on 1)

The system of claim 1, wherein the image patch comprises a text finger print of the printed document having at least two lines that is usable for pattern matching.

### Claim 7 (depends on 1)

The system of claim 1, wherein the image patch uniquely identifies a location in the printed document.

### Claim 8 (depends on 7)

The system of claim 7, wherein the image patch comprises a text finger print of the printed document having a portion of a first line and a portion of an adjacent second line.

### Claim 9 (depends on 2)

The system of claim 2, wherein the action comprises text to be presented or one or more commands to be executed, and wherein the index is a link between the text to be presented or one or more commands to be executed and the electronic representation of the printed document.

### Claim 10

A computer-implemented method for creating a mixed media reality document data structure for interacting with printed documents and electronic documents, the method comprising:
creating the mixed media reality document data structure, comprising:
a representation of a printed document comprising a matchable portion for use in pattern matching to a corresponding first location within a captured image patch of the printed document, wherein the image patch comprises a portion less than an entire page of the document;
an electronic representation of the printed document;
and an index correlating the matchable portion of the first representation with a second location in the electronic document;
wherein creating the mixed media reality document data structure further comprises:
generating the representation of the printed document;
indexing the generated representation of the printed document;
capturing the image patch from the printed document;
comparing the image patch to the generated representation; and
performing an action if the image patch is similar to the generated representation.

### Claim 11 (depends on 10)

The method of claim 10, wherein the generating the representation of the printed document comprises automatically capturing a text layout of the printed document.

### Claim 12 (depends on 11)

The method of claim 11, wherein the automatically capturing is done by one from a group of a printed document capture module, an operating system, a print driver, a scanner, a file transfer mechanism.

### Claim 13 (depends on 10)

The method of claim 10, wherein the generated representation is one from the group of a representation of the entire printed document document, a representation of a patch of text, a representation of a single word, a representation of a section of an image, and any other representation of a matchable portion of the printed document.

### Claim 14 (depends on 10)

The method of claim 10, wherein indexing the generated representation includes association of one from the group of the index, an electronic representation of the document, a second media type, and an action, with the generated representation.

### Claim 15 (depends on 10)

The method of claim 10, wherein indexing the generated representation includes one from the group of specifying x-y locations of characters of the printed document, specifying x-y locations of words of the printed document, specifying x-y locations of an image or a portion of an image in a printed document, performing an OCR operation and specifying x-y locations of characters and/or words, performing feature extraction and specifying x-y locations of characters and/or words.

### Claim 16 (depends on 10)

The method of claim 10, wherein capturing the image patch comprises one from the group of scanning a patch of text, scanning a patch of text and preprocessing to determine the likelihood that an intended feature description is extracted correctly, scanning a machine-readable identifier, inputting data that identifies the document, receiving email with the document attached, receiving the document by file transfer, scanning a portion of an image, and inputting the document.

### Claim 17 (depends on 10)

The method of claim 10, wherein comparing the image patch to the generated representation further comprises performing pattern matching between the image patch and the generated representation.

### Claim 18 (depends on 17)

The method of claim 17, wherein the pattern matching is performed by comparing a document fingerprint of the image patch to a document fingerprint of the generated first representation.

### Claim 19 (depends on 18)

The method of claim 18, wherein comparing the document fingerprint includes extracting features from a captured image, composing descriptors from those features, and looking up a document and patch that contain a percentage of those descriptors.

### Claim 20 (depends on 10)

The method of claim 10, wherein the action is one from the group of retrieving information, writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, and input of a command to another system.