# US 2014/0192210 A1 · Mobile device based text detection and tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0192210 A1
- Document ID: US20140192210A1
- Status: Published application
- Kind: A1
- Application number: 14/021,337
- Filed: 2013-09-09
- Published: 2014-07-10
- Earliest claimed: 2013-01-04
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Michael GERVAUTZ; Jieun KIM; Per O. NIELSEN; Roy LAWRENCE ASHOK INIGO; Qi PAN; Romain TALLONNEAU
- Web page: https://patentgrep.com/patent/US20140192210A1

## CPC

- G06V 30/224: of printed characters having additional code marks or containing code marks
- G06F 3/005: {Input arrangements through a video camera}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/74: {involving reference images or patches}
- G06V 30/142: using hand-held instruments; Constructional details of the instruments
- G10L 13/00: Speech synthesis; Text to speech systems
- H04N 1/00204: {with a digital computer or a digital computer system, e.g. an internet server (programmed control between transmitter and receiver or between image input and image output device H04N1/32561)}
- G06T 2207/30244: Camera pose
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Embodiments disclosed pertain to mobile device based text detection and tracking. In some embodiments, a first reference frame is obtained by performing Optical Character Recognition (OCR) on an image frame captured by a camera to locate and recognize a first text block. A subsequent image frame may be selected from a set of subsequent image frames based on parameters associated with the selected subsequent image and a second reference frame may be obtained by performing OCR on the selected subsequent image frame to recognize a second text block. A geometric relationship between the first and second text blocks is determined based on a position of the first text block in the second reference frame and a pose associated with the second reference frame.

## Claims

### Claim 1

A method on a mobile station (MS), the method comprising:
obtaining a first reference frame by performing Optical Character Recognition (OCR) on an image frame captured by a camera on the MS to locate and recognize a first text block;
selecting a subsequent image frame from a set of subsequent image frames, based on parameters associated with the selected subsequent image frame;
obtaining a second reference frame by performing OCR on the selected subsequent image frame to recognize a second text block; and
determining a geometric relationship between the first and second text blocks based, at least in part, on a position of the first text block in the second reference frame and a camera pose associated with the second reference frame.

### Claim 2 (depends on 1)

The method of claim 1, further comprising assembling the first and second text blocks in a sequence based on the geometric relationship between the first and second text blocks.

### Claim 3 (depends on 2)

The method of claim 2, wherein the geometric relationship between the first and second text blocks is based, at least in part, on a frame of reference associated with a medium on which the text blocks appear.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
providing the assembled sequence of first and second text blocks as input to a text-to-speech application.

### Claim 5 (depends on 1)

The method of claim 1, wherein selecting the subsequent image frame further comprises:
computing camera poses for the set of subsequent image frames, each camera pose associated with a distinct subsequent image frame and determined based, at least in part, on aligning the associated subsequent image frame with the first reference frame, and
determining, based, at least in part, on the computed camera poses, parameters associated with corresponding image frames in the set of subsequent image frames.

### Claim 6 (depends on 5)

The method of claim 5, wherein the aligning is performed using Efficient Second-order Minimization (ESM).

### Claim 7 (depends on 6)

The method of claim 6, wherein the ESM operates on a lower resolution version of the associated subsequent image frame.

### Claim 8 (depends on 5)

The method of claim 5, wherein computing camera poses for the set of subsequent image frames further comprises:
generating a tracking target comprising image patches obtained by identifying a plurality of feature points in the first reference frame, and
determining a location of the tracking target in a subsequent image frame in the set based on a correspondence of image patches between the first reference frame and the subsequent image frame, and
computing a camera pose associated with the subsequent image frame based, at least in part, on the location of the tracking target in the subsequent image frame.

### Claim 9 (depends on 8)

The method of claim 8, wherein the feature points are based on natural features in the first reference frame.

### Claim 10 (depends on 8)

The method of claim 8, wherein individual feature points are assigned weights and feature points over the first text block are assigned a greater weight relative to feature points located elsewhere in the first reference frame.

### Claim 11 (depends on 8)

The method of claim 8, wherein generation of the tracking target is performed substantially in parallel with the aligning of the associated subsequent image frame with the first reference frame.

### Claim 12 (depends on 1)

The method of claim 1, wherein the first reference frame and the set of subsequent image frames are markerless.

### Claim 13 (depends on 1)

The method of claim 1, wherein the parameters comprise at least one of:
a percentage of overlap area between the selected subsequent image frame and the first reference frame, or
a fraction of the first text block visible in the selected subsequent image frame, wherein the fraction is determined as a ratio of an area comprising a visible portion of the first text block in the selected subsequent image frame to a total area of the first text block, or
a magnitude of rotation of the selected subsequent image frame relative to the first reference frame, or
a magnitude of translation of the selected subsequent image frame relative to the first reference frame.

### Claim 14 (depends on 1)

The method of claim 1, wherein the camera pose is determined in 6 Degrees of Freedom (6-DoF), wherein the camera is fronto-parallel to a planar medium comprising the text blocks.

### Claim 15 (depends on 1)

The method of claim 1, wherein the method is invoked by an Augmented Reality (AR) application.

### Claim 16 (depends on 15)

The method of claim 15, wherein a virtual object is placed by the AR application over the first and second text blocks.

### Claim 17 (depends on 16)

The method of claim 16, wherein the virtual object comprises translated text from the first and second text blocks, wherein the translated text is in a language different from a language used to express the first and second text blocks.

### Claim 18

A mobile station (MS) comprising:
a camera configured to capture a first image frame and a set of subsequent image frames, and
a processor coupled to the camera, the processor comprising:
a word recognition module configured to:
obtain a first reference frame by performing Optical Character Recognition (OCR) on the first image frame to locate and recognize a first text block,
select a subsequent image frame from the set of subsequent image frames, based on parameters associated with the selected subsequent image frame; and
obtain a second reference frame by performing OCR on the selected subsequent image frame to recognize a second text block; and
a text assembler module configured to determine a geometric relationship between the first and second text blocks based, at least in part, on a position of the first text block in the second reference frame and a camera pose associated with the second reference frame.

### Claim 19 (depends on 18)

The MS of claim 18, wherein the text assembler module is further configured to:
assemble the first and second text blocks in a sequence based on the geometric relationship between the first and second text blocks.

### Claim 20 (depends on 19)

The MS of claim 19, wherein the text assembler module is further configured to:
provide the assembled sequence of first and second text blocks as input to a text-to-speech application.

### Claim 21 (depends on 18)

The MS of claim 18, wherein the processor further comprises a tracking module operationally coupled to the word recognition module, the tracking module configured to:
compute camera poses for the set of subsequent image frames, each camera pose associated with a distinct subsequent image frame and determined based, at least in part, on aligning the associated subsequent image frame with the first reference frame, and
determine, based, at least in part, on the computed camera poses, parameters associated with corresponding image frames in the set of subsequent image frames.

### Claim 22 (depends on 21)

The MS of claim 21, wherein the tracking module is further configured to perform the aligning using Efficient Second-order Minimization (ESM).

### Claim 23 (depends on 22)

The MS of claim 22, wherein the ESM operates on a lower resolution version of the associated subsequent image frame.

### Claim 24 (depends on 21)

The MS of claim 21, wherein to compute camera poses for the set of subsequent image frames, the tracking module is further configured to:
generate a tracking target comprising image patches obtained by identifying a plurality of feature points in the first reference frame, and
determine a location of the tracking target in a subsequent image frame in the set based on a correspondence of image patches between the first reference frame and the subsequent image frame, and
compute a camera pose associated with the subsequent image frame based, at least in part, on the location of the tracking target in the subsequent image frame.

### Claim 25 (depends on 24)

The MS of claim 24, wherein the feature points are based on natural features in the first reference frame.

### Claim 26 (depends on 24)

The MS of claim 24, wherein the tracking module is configured to assign weights to individual feature points such that feature points over the first text block are assigned a greater weight relative to feature points located elsewhere in the first reference frame.

### Claim 27 (depends on 24)

The MS of claim 24, wherein the tracking module is configured to generate the tracking target substantially in parallel with the aligning of the associated subsequent image frame with the first reference frame.

### Claim 28 (depends on 18)

The MS of claim 18, wherein the first reference frame and the set of subsequent image frames captured by the camera are markerless.

### Claim 29 (depends on 18)

The MS of claim 18, wherein the parameters comprise at least one of:
a percentage of overlap area between the selected subsequent image frame and the first reference frame, or
a fraction of the first text block visible in the selected subsequent image frame, wherein the fraction is determined as a ratio of an area comprising a visible portion of the first text block in the selected subsequent image frame to a total area of the first text block, or
a magnitude of rotation of the selected subsequent image frame relative to the first reference frame, or
a magnitude of translation of the selected subsequent image frame relative to the first reference frame.

### Claim 30

An apparatus comprising:
imaging means for capturing a sequence of image frames,
means for obtaining a first reference frame by performing Optical Character Recognition (OCR) on an image frame in the sequence of image frames to locate and recognize a first text block,
means for selecting a subsequent image frame from the sequence of image frames, the selection based on parameters associated with the selected subsequent image frame,
means for obtaining a second reference frame by performing OCR on the selected subsequent image frame to recognize a second text block, and
means for determining a geometric relationship between the first and second text blocks based, at least in part, on a position of the first text block in the second reference frame and a pose of the imaging means associated with the second reference frame.

### Claim 31 (depends on 30)

The apparatus of claim 30, further comprising:
means for assembling the first and second text blocks in a sequence based on the geometric relationship between the first and second text blocks.

### Claim 32 (depends on 31)

The apparatus of claim 31, further comprising:
means for providing the assembled sequence of first and second text blocks as input to a text-to-speech application.

### Claim 33 (depends on 30)

The apparatus of claim 30, wherein the means for selecting a subsequent image frame comprises:
means for computing poses of the imaging means for the image frames in the sequence of image frames, each computed pose of the imaging means associated with a distinct image frame and determined, at least in part, by aligning the associated image frame with the first reference frame, and
means for determining based, at least in part, on the computed poses of the imaging means, parameters associated with corresponding image frames in the sequence of image frames.

### Claim 34 (depends on 33)

The apparatus of claim 33, wherein the means for computing poses of the imaging means comprises:
means for generating a tracking target comprising image patches obtained by identifying a plurality of feature points in the first reference frame, and
means for determining a location of the tracking target in a subsequent image frame in the sequence of image frames based on a correspondence of image patches between the first reference frame and the subsequent image frame, and
means for computing a camera pose associated with the subsequent image frame based, at least in part, on the location of the tracking target in the subsequent image frame.

### Claim 35 (depends on 34)

The apparatus of claim 34, wherein individual feature points are assigned weights and feature points over the first text block are assigned a greater weight relative to feature points located elsewhere in the first reference frame.

### Claim 36 (depends on 34)

The apparatus of claim 34, wherein the means for generating the tracking target operates substantially in parallel with the aligning of the associated image frame with the first reference frame.

### Claim 37 (depends on 30)

The apparatus of claim 30, wherein the image frames in the sequence of image frames captured by the imaging means are markerless.

### Claim 38 (depends on 30)

The apparatus of claim 30, wherein the parameters comprise at least one of:
a percentage of overlap area between the selected subsequent image frame and the first reference frame, or
a fraction of the first text block visible in the selected subsequent image frame, or
a magnitude of rotation of the selected subsequent image frame relative to the first reference frame, or
a magnitude of translation of the selected subsequent image frame relative to the first reference frame.

### Claim 39

A non-transitory computer-readable medium comprising instructions, which, when executed by a processor, perform a method on a Mobile Station (MS), the method comprising:
obtaining a first reference frame by performing Optical Character Recognition (OCR) on an image frame captured by a camera on the MS to locate and recognize a first text block;
selecting a subsequent image frame from a set of subsequent image frames, the selection based on parameters associated with the selected subsequent image frame;
obtaining a second reference frame by performing OCR on the selected subsequent image frame to recognize a second text block; and
determining a geometric relationship between the first and second text blocks based, at least in part, on a position of the first text block in the second reference frame and a camera pose associated with the second reference frame.