# US 2008/0186255 A1 · Systems and methods for data annotation, recordation, and communication > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2008/0186255 A1 - Document ID: US20080186255A1 - Status: Published application - Kind: A1 - Application number: 11/952,896 - Filed: 2007-12-07 - Published: 2008-08-07 - Earliest claimed: 2006-12-07 - Inventors: Philip R. Cohen; David McGee - Web page: https://patentgrep.com/patent/US20080186255A1 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G02B 27/017: {Head mounted} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0321: {by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet (constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037)} - G06F 3/0325: {using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image} - G06F 3/03545: {Pens or stylus} - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G02B 27/01: Head-up displays - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract Systems, devices, and methods to provide tools enhance the tactical or strategic situation awareness of on-scene and remotely located personnel involved with the surveillance of a region-of-interest using field-of-view sensory augmentation tools. The sensory augmentation tools provide updated, visual, text, audio, and graphic information associated with the region-of-interest adjusted for the positional frame of reference of the on-scene or remote personnel viewing the region-of-interest, map, document or other surface. Annotations and augmented reality graphics projected onto and positionally registered with objects or regions-of-interest visible within the field of view of a user looking through a see through monitor may select the projected graphics for editing and manipulation by sensory feedback from the viewer. ## Claims ### Claim 1 A method for viewing information, the method comprising: generating a symbol using a digital pen in proximity to an information-carrying-surface having fiducial reference markers operable for orienting a image capturing device; transmitting the symbol to a processor; converting the symbol to a realistic depiction representative of the symbol; and displaying the realistic depiction representative of the symbol on a display device. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the symbol includes drawing the symbol on the information-carrying-surface with the digital pen. ### Claim 3 (depends on 1) The method of claim 1, wherein generating the symbol includes making a hand gesture within a field of view of the image capturing device and in proximity to the information-carrying-surface. ### Claim 4 (depends on 1) The method of claim 1, further comprising: recording and recognizing voice information while generating the symbol in proximity to the information-carrying-surface. ### Claim 5 (depends on 1) The method of claim 1, wherein the information-carrying-surface includes a micro pattern having micro-dots printed on at least one side of the information-carrying-surface. ### Claim 6 (depends on 5) The method of claim 5, wherein the micro pattern is registered onto the information-carrying-surface for permitting motion tracking of the digital pen relative to a location of the digital pen with respect to the information-carrying-surface. ### Claim 7 (depends on 1) The method of claim 1, wherein transmitting the symbol to a processor includes transmitting data associated with the symbol from the digital pen. ### Claim 8 (depends on 1) The method of claim 1, wherein the fiducial reference markers include objects viewable within a field of view of the image capturing device, the objects selected from the group consisting of graphical objects, alpha-numeric objects, geometric objects, symbolic objects, hand-written objects, printed objects, reflective objects, and contoured objects. ### Claim 9 (depends on 1) The method of claim 1, wherein the fiducial reference markers include objects viewable within the field of view of the image capturing device, the objects selected from the group consisting of mirrors positioned along at least a portion of the periphery of the information-carrying surface, motion sensors positioned along at least a portion of the periphery of the information-carrying surface, and sound sensors positioned along at least a portion of the periphery of the information surface. ### Claim 10 (depends on 1) The method of claim 1, wherein the fiducial reference markers are arranged relative to the information-carrying-surface to provide information related to a spatial orientation of the image capturing device. ### Claim 11 (depends on 1) The method of claim 1, wherein transmitting the symbol to the processor includes sending the signal over a wireless communication link. ### Claim 12 (depends on 1) The method of claim 1, wherein displaying the realistic depiction representative of the symbol on the display device includes displaying the realistic depiction representative of the symbol on a computer monitor. ### Claim 13 (depends on 1) The method of claim 1, wherein displaying the realistic depiction representative of the symbol on the display device includes displaying the realistic depiction representative of the symbol on a head-worn display device. ### Claim 14 (depends on 13) The method of claim 13, wherein displaying the realistic depiction representative of the symbol on the head-worn display device includes displaying the realistic depiction representative of the symbol on a substantially transparent screen. ### Claim 15 (depends on 1) The method of claim 1, wherein displaying the realistic depiction of the object on the substantially transparent screen includes permitting a viewer of the realistic depiction representative of the symbol to view the realistic depiction while maintaining a field of view beyond the substantially transparent screen. ### Claim 16 (depends on 1) The method of claim 1, wherein the image capturing device includes a complementary metal-oxide-semiconductor (CMOS) image sensor. ### Claim 17 (depends on 1) The method of claim 1, wherein the image capturing device includes an image sensor having a charge coupled device (CCD). ### Claim 18 (depends on 1) The method of claim 1, further comprising: interacting with the realistic depiction representative of the symbol after the realistic depiction is displayed. ### Claim 19 (depends on 1) The method of claim 1, wherein displaying the realistic depiction representative of the symbol on the display device includes projecting the realistic depiction representative of the symbol on the information carrying surface. ### Claim 20 (depends on 17) The method of claim 17, wherein interacting with the realistic depiction includes manipulating the depiction using gestures and voice commands.