# US 8,787,706 B2 · Acquisition of a user expression and an environment of the expression > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,787,706 B2 - Document ID: US8787706B2 - Status: Granted - Kind: B2 - Application number: 11/097,980 - Filed: 2005-03-31 - Published: 2006-09-21 - Granted: 2014-07-22 - Earliest claimed: 2005-03-18 - Assignees: The Invention Science Fund I, LLC - Applicants: Alexander J. Cohen; Edward K. Y. Jung; Royce A. Levien; Robert W. Lord; Mark A. Malamud; John D. Rinaldo, Jr. - Inventors: Alexander J. Cohen; Edward K. Y. Jung; Royce A. Levien; Robert W. Lord; Mark A. Malamud; John D. Rinaldo, Jr. - Web page: https://patentgrep.com/patent/US8787706B2 ## CPC - G06F 3/002: {Specific input/output arrangements not covered by G06F3/01 - G06F3/16 (other optical apparatus G02B27/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/03545: {Pens or stylus} - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 40/174: Form filling; Merging - G06V 30/32: Digital ink - H04L 67/04: specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Embodiments include an apparatus, device, system, and method. In an embodiment, a human wearable apparatus is provided. The apparatus includes an annotation environment capture device operable to generate information indicative of a recognizable aspect of an item and an annotating device operable to generate information indicative of a user expression associated with the recognizable aspect of the item. ## Claims ### Claim 1 A human wearable apparatus, the apparatus comprising: a) an annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item while at least one user's eye gazes at the item, the annotation environment capture device being further operable to generate information indicative of content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, at least a portion of the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion; and b) an annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device includes at least a head mountable annotation environment device. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the annotating device includes at least a head mountable annotating device. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes; an annotation environment capture device operable to capture at least one parameter of an environment that includes at least the machine-recognizable aspect of an item. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine recognizable aspect of an item includes: an annotation environment capture device operable to capture an optical representation of an environment that includes at least the machine-recognizable aspect of an item. ### Claim 6 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to capture an audio representation of an environment that includes at least the machine-recognizable aspect of an item. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a machine-recognizable radio frequency identification signal corresponding to an aspect of an item. ### Claim 8 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of an optically-recognizable aspect of an item. ### Claim 9 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a human recognizable aspect of an item. ### Claim 10 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a recognizable native text of an item. ### Claim 11 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a recognizable visual aspect of an item. ### Claim 12 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a recognizable image aspect of an item. ### Claim 13 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a recognizable shape aspect of an item. ### Claim 14 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a recognizable dimensional aspect of an item. ### Claim 15 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a recognizable human readable content. ### Claim 16 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a recognizable aspect of an item in response to an input received from the user. ### Claim 17 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: an annotation environment capture device operable to generate information indicative of a recognizable aspect of an item in response to an input responsive to a gesture by the user. ### Claim 18 (depends on 1) The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes: a module operable to recognize data indicative of the recognizable aspect of an item. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein the module operable to recognize data indicative of the machine-recognizable aspect of an item includes: an artificial intelligence module operable to recognize data indicative of the recognizable aspect of an item. ### Claim 20 (depends on 18) The apparatus of claim 18, wherein the module operable to recognize data indicative of the machine-recognizable aspect of an item includes: a pattern recognition module operable to recognize data indicative of the recognizable aspect of an item. ### Claim 21 (depends on 1) The apparatus of claim 1, wherein the annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item includes: an annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item, wherein the user expression includes at least one of a hand formed user expression, a verbal user expression, a preformed user expression, or a gestural user expression. ### Claim 22 (depends on 1) The apparatus of claim 1, further comprising: c) a communications device. ### Claim 23 (depends on 1) The apparatus of claim 1, further comprising: c) a storage device. ### Claim 24 A head mountable apparatus, the apparatus comprising: a) an annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item while at least one user's eye gazes at the item, the annotation environment capture device being further operable to generate information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in each of the at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, at least a portion of the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion; and b) an annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 25 (depends on 24) The apparatus of claim 24, further comprising: c) a communications device. ### Claim 26 (depends on 24) The apparatus of claim 24, further comprising: c) a storage device. ### Claim 27 A non-transitory computer-readable medium having a set of instructions for a computer-implemented method, the set of instructions comprising: a) one or more instructions for generating in a portable apparatus a first information indicative of a machine-recognizable aspect of an item while at least one user's eye gazes at the item, wherein the generating the first information as responsive to a user direction to capture the machine-recognizable aspect of the item; b) one or more instructions for generating in the portable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item; and c) one or more instructions for generating in the portable apparatus information indicative of a content portion of an item, the generation of the information indicative of content existing in at least two planes and including at least detection of detection of at least a portion of content in at least two planes substantially simultaneously the information indicative of the content portion including at least contextual information, wherein at least a portion of the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion. ### Claim 28 (depends on 27) The computer-readable medium of claim 27, wherein the one or more instructions for generating in a portable apparatus a first information indicative of a machine-recognizable aspect of an item includes: one or more instructions for generating in a portable head mountable apparatus a first information indicative of the machine-recognizable aspect of an item. ### Claim 29 (depends on 28) The computer-readable medium of claim 28, wherein the one or more instructions for generating in the portable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes: one or more instructions for generating in the portable head mountable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 30 (depends on 27) The computer-readable medium of claim 27, further comprising: c) one or more instructions for saving at least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 31 (depends on 27) The computer-readable medium of claim 27, further comprising: c) one or more instructions for transmitting at least one of the first information indicative of a machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 32 A wearable apparatus, the comprising: a) a context module operable to generate a first information indicative of a machine-recognizable aspect of an item while at least one user's eye gazes at the item, the context module being further operable to generate information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion includes at least contextual information, at least a portion of the contextual information being acquired by ultrasound and being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion; and b) an annotation module operable to generate a second information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 33 (depends on 32) The apparatus of claim 32, further comprising: c) a communications module operable to transmit at least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 34 (depends on 32) The apparatus of claim 32, further comprising: c) a storage module operable to save at least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 35 A method comprising: a) generating in a portable apparatus a first information indicative of a machine-recognizable aspect of an item while at least one user's eye gazes at the item; b) generating in the portable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item, wherein the user expression is recognized in response to the portable apparatus recognizing the machine-recognizable aspect of the item; and c) generating in the portable apparatus information indicative of a content portion of an item, the generating of the information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in each of the at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, wherein at least a portion of the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion. ### Claim 36 (depends on 35) The method of claim 35, wherein generating in a portable apparatus a first information indicative of a machine-recognizable aspect of an item includes: generating in a portable head mountable apparatus a first information indicative of the machine-recognizable aspect of an item. ### Claim 37 (depends on 36) The method of claim 36, wherein generating in the portable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes: generating in the portable head mountable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 38 (depends on 35) The method of claim 35, further comprising: c) saving at least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 39 (depends on 35) The method of claim 35, further comprising: c) transmitting least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item. ### Claim 40 A computer-implemented method comprising: a) tracking a gaze of at least one user's eye; b) acquiring an image of a machine-recognizable aspect of an item corresponding to the gaze of the at least one user's eye; c) generating a first information indicative of the machine-recognizable aspect of an item corresponding to the gaze of the at least one user's eye while a user views the item; d) generating a second information indicative of a user expression associated with the machine-recognizable aspect of an item corresponding to the gaze of the at least one user's eye; and e) generating information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion. ### Claim 41 (depends on 40) The method of claim 40, further comprising: e) receiving a signal indicative of the user expression associated with the machine-recognizable aspect of the item. ### Claim 42 (depends on 40) The method of claim 40, wherein acquiring an image of a machine-recognizable aspect of the item corresponding to the gaze of the at least one user's eye includes: acquiring an image of a machine-recognizable aspect of the item corresponding to the gaze of the at least one user's eye in response to a received user input. ### Claim 43 (depends on 40) The method of claim 40, wherein generating a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes: generating a second information indicative of a user verbal expression associated with the machine-recognizable aspect of the item. ### Claim 44 (depends on 40) The method of claim 40, wherein generating a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes: generating a second information indicative of a user hand formed expression associated with the machine-recognizable aspect of the item. ### Claim 45 (depends on 40) The method of claim 40, wherein generating a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes: generating a second information indicative of a user gesture associated with the machine-recognizable aspect of the item. ### Claim 46 (depends on 40) The method of claim 40, wherein generating information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion includes: generating, in a head mountable apparatus, information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion. ### Claim 47 A head mountable apparatus, the apparatus comprising: a) a tracking module operable to track a gaze of at least one user's eye; b) a sensor module operable to acquire an image of a machine-recognizable aspect of the item corresponding to the gaze of the at least one user's eye; c) a context module operable to generate a first information indicative of the machine-recognizable aspect of an item corresponding to the gaze of the at least one user's eye while a user views the item; d) an annotation module operable to generate a second information indicative of a user expression associated with the machine-recognizable aspect of the item; and e) wherein the context module is further operable to generate information indicative of a content portion of an item, at least a portion of the information being indicative of content existing in at least two planes and including at least detection of at least a portion of the content in at least two planes substantially simultaneously, and wherein the content portion includes at least an incidental element configured to aid in distinguishing the item, wherein the incidental element is not readable by a human, the information indicative of the content portion includes at least contextual information associated with a handwriting movement on at least one of the at least two planes proximate to the content portion. ### Claim 48 (depends on 47) The apparatus of claim 47, further comprising: a receiver module operable to receive a signal indicative of the user expression associated with the machine-recognizable aspect of the item. ### Claim 49 (depends on 1) The apparatus of claim 1, wherein the annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item comprises: an annotating device operable to generate information indicative of a visual handwritten expression associated with the machine-recognizable aspect of the item. ### Claim 50 (depends on 1) The apparatus of claim 1, wherein the annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item comprises: an annotating device operable to generate information indicative of a spatial handwritten expression associated with the machine-recognizable aspect of the item. ### Claim 51 (depends on 1) The apparatus of claim 1, wherein the annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item comprises: an annotating device operable to generate information indicative of an audible user expression associated with the machine-recognizable aspect of the item. ### Claim 52 (depends on 1) The apparatus of claim 1, wherein the content portion includes an incidental element, wherein the incidental element includes at least an indication of a torn page. ### Claim 53 (depends on 1) The apparatus of claim 1, wherein the content portion includes an incidental element, the incidental element includes at least an indication of a stain correlated with the content portion. ### Claim 54 (depends on 1) The apparatus of claim 1, wherein the information indicative of the content portion includes at least contextual information, wherein the contextual information is acquired from another source. ### Claim 55 (depends on 1) The apparatus of claim 1, wherein at least a portion of the contextual information comprises: contextual information acquired by ultrasound.