# US 8,279,456 B2 · Handheld display device having processor for rendering display output with real-time virtual transparency and form-filling option

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,279,456 B2
- Document ID: US8279456B2
- Status: Granted
- Kind: B2
- Application number: 13/284,932
- Filed: 2011-10-30
- Published: 2012-02-23
- Granted: 2012-10-02
- Earliest claimed: 1999-12-01
- Assignees: Silverbrook Research Pty Ltd
- Inventors: Paul Lapstun; Kia Silverbrook; Tobin Allen King
- Web page: https://patentgrep.com/patent/US8279456B2

## CPC

- B41J 3/445: {Printers integrated in other types of apparatus, e.g. printers integrated in cameras}
- G06F 1/1626: {with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]}
- G06F 1/1656: {Details related to functional adaptations of the enclosure, e.g. to provide protection against EMI, shock, water, or to host detachable peripherals like a mouse or removable expansions units like PCMCIA cards, or to provide access to internal components for maintenance or to removable storage supports like CDs or DVDs, or to mechanically mount accessories (mounting of accessories to a computer display G06F1/1607; display hoods G06F1/1603; cooling arrangements for portable computers G06F1/203)}
- G06F 1/1684: {Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675}
- G06F 1/1688: {the I/O peripheral being integrated loudspeakers}
- G06F 3/0321: {by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet (constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037)}
- G06F 3/03545: {Pens or stylus}
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06Q 40/00: Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q 40/08: Insurance
- H04M 1/27475: using interactive graphical means or pictorial representations
- H04M 1/2755: by optical scanning
- H04M 1/72409: by interfacing with external accessories (hands-free H04M1/60)
- H04M 1/72445: for supporting Internet browser applications
- H04M 3/42: Systems providing special services or facilities to subscribers (specially adapted for wireless communication networks H04W4/00)
- H04N 1/00204: {with a digital computer or a digital computer system, e.g. an internet server (programmed control between transmitter and receiver or between image input and image output device H04N1/32561)}
- H04N 1/00244: {with a server, e.g. an internet server (fax-servers or the like for store and forward H04N1/324)}
- H04N 1/00307: {with a mobile telephone apparatus (constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00)}
- H04N 1/00326: {with a data reading, recognizing or recording apparatus, e.g. with a bar-code apparatus (arrangements for the associated working of recording or reproducing apparatus with related apparatus G11B31/00)}
- H04N 1/00392: {Other manual input means, e.g. digitisers or writing tablets}
- H04N 1/00968: {by scanning marks on a sheet (mark-sheet input means H04N1/00355; sensing record carriers G06K7/00)}
- H04N 1/32122: {in a separate device, e.g. in a memory or on a display separate from image data}
- H04N 1/32778: {using a mark-sheet or machine-readable code request}
- H04N 5/445: for displaying additional information (H04N5/50 takes precedence)
- H04N 5/76: Television signal recording
- H04N 7/163: {by receiver means only}
- H04N 19/60: using transform coding
- H04N 19/61: in combination with predictive coding
- H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)}
- H04N 21/422: Input-only peripherals {, i.e. input devices connected to specially adapted client devices}, e.g. global positioning system [GPS] {(input devices also receiving signals from specially adapted client devices H04N21/4104)}
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G06F 2200/1614: Image rotation following screen orientation, e.g. switching from landscape to portrait mode
- G06F 2200/1637: Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
- G06F 2203/0384: Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
- H04M 1/72412: using two-way short-range wireless interfaces
- H04M 1/72442: for playing music files
- H04M 7/0015: {First party call control architectures}
- H04M 2250/12: including a sensor for measuring a physical value, e.g. temperature or motion
- H04N 1/32133: {on the same paper sheet, e.g. a facsimile page header}
- H04N 5/765: Interface circuits between an apparatus for recording and another apparatus
- H04N 5/783: Adaptations for reproducing at a rate different from the recording rate
- H04N 5/907: using static stores, e.g. storage tubes or semiconductor memories (H04N5/91 takes precedence)
- H04N 21/478: Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N 2201/0082: Image hardcopy reproducer (H04N2201/0091 - H04N2201/0094 take precedence)
- H04N 2201/3226: of identification information or the like, e.g. ID code, index, title, part of an image, reduced-size image
- H04N 2201/3247: Data linking a set of images to one another, e.g. sequence, burst or continuous capture mode
- H04N 2201/3264: of sound signals
- H04N 2201/3269: of machine readable codes or marks, e.g. bar codes or glyphs
- H04N 2201/3274: Storage or retrieval of prestored additional information
- Y10S 707/99931: Database or file accessing
- Y10T 428/187: Continuous sectional layer
- B41J: TYPEWRITERS; SELECTIVE PRINTING MECHANISMS, i.e. MECHANISMS PRINTING OTHERWISE THAN FROM A FORME; CORRECTION OF TYPOGRAPHICAL ERRORS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10T: TECHNICAL SUBJECTS COVERED BY FORMER US CLASSIFICATION

## Abstract

A handheld display device for interacting with printed content and filling in a form. The device includes: an optical sensor for imaging an area of a printed substrate containing the printed content and generating image data; an opaque touch-sensitive display screen for displaying display output to a user; a processor configured for: monitoring the identity of the substrate and the position of the device relative to the substrate using the image data; retrieving display data; rendering the display output to the display screen; and interpreting user input to a displayed form via the touch-sensitive display screen. The opaque touch-sensitive display screen has real-time virtual transparency through the screen to the printed content from a user's perspective to provide a virtual window to the printed content.

## Claims

### Claim 1

A handheld display device for interacting with printed content and filling in a form, said device comprising:
an optical sensor for imaging an area of a printed substrate containing the printed content and generating image data;
an opaque touch-sensitive display screen for displaying display output to a user; and
a processor configured for:
monitoring the identity of the substrate and the position of the device relative to the substrate using the image data;
retrieving display data based on the position of the device relative to the substrate;
rendering the display output to the touch-sensitive display screen, said display output being at least partially based on the retrieved display data, said display output including a displayed form; and
interpreting user input to the displayed form via the touch-sensitive display screen,
wherein the opaque touch-sensitive display screen has real-time virtual transparency through the screen to the printed content from a user's perspective to provide a virtual window to the printed content.

### Claim 2 (depends on 1)

The handheld display device of claim 1, wherein the user input to the displayed form employs a non-marking stylus.

### Claim 3 (depends on 1)

The handheld display device of claim 1, which is a mobile phone or a personal digital assistant.

### Claim 4 (depends on 1)

The handheld display device of claim 1, further comprising a digital transceiver.

### Claim 5 (depends on 4)

The handheld display device of claim 4, wherein said digital transceiver is at least one of: a short-range wireless transceiver and a mobile phone transceiver.

### Claim 6

An interactive application is stored on a non-transitory computer readable medium for execution on a handheld display device having an image sensor, a processor and an opaque touch-sensitive display screen, the interactive application being configured to operatively control the device so as to perform the steps of: monitoring the identity of the substrate and the position of the device relative to the substrate using the image data; retrieving display data based on the position of the device relative to the substrate; rendering display output to the touch-sensitive display screen, said display output being at least partially based on the retrieved display data, said display output including a displayed form; and interpreting user input to the displayed form via the touch-sensitive display screen, wherein the interactive application is configured to operatively control the device such that the opaque touch-sensitive display screen has real-time virtual transparency through the screen to the printed content from a user's perspective to provide a virtual window to the printed content.

### Claim 7 (depends on 6)

The interactive application of claim 6, wherein the displayed form comprises one or more fill-in fields.

### Claim 8 (depends on 6)

The interactive application of claim 6, which is configured to operatively control the device such that the displayed form is snapped to the display output in response to a user command, whereby the display output comprises the displayed form irrespective of a position of the device relative to the substrate.

### Claim 9

A method of interacting with printed content and filling in a form using a handheld display device, said method comprising the steps of:
imaging an area of a printed substrate and generating image data using an image sensor of the handheld display device, the imaged area of the printed substrate containing the printed content;
monitoring the identity of the substrate and the position of the device relative to the substrate using the image data;
retrieving display data based on the position of the device relative to the substrate;
rendering display output to the touch-sensitive display screen, said display output being at least partially based on the retrieved display data, said display output including a displayed form; and
interpreting user input to the displayed form via the touch-sensitive display screen, wherein the opaque touch-sensitive display screen has real-time virtual transparency through the screen to the printed content from a user's perspective to provide a virtual window to the printed content.

### Claim 10 (depends on 9)

The method of claim 9, wherein the user input to the displayed form employs a non-marking stylus.

### Claim 11 (depends on 9)

The method of claim 9, further comprising the step of:
snapping the displayed form to the display output, such that the display output comprises the displayed form irrespective of a position of the device relative to the substrate.
