Augmenting and not replacing paper based work practice via multi-modal interaction

A system for augmenting a paper-based work practice such that a combination physical record formed from a combination of a primary physical document and one or more secondary physical documents is retained in original form to provide a backup and an integrated electronic version, that includes electronic inputs including by electronic capturing some or all of the combination physical record, is prepared and made accessible for analysis and communication, said augmenting system comprising:

• a touch board sized and shaped to accommodate the primary physical document positioned thereon so that the primary physical document has positions relative to said touch board;
• a keyboard;
• a microphone;
• a computer system to which said touch board and said microphone are cooperatively connected such that, after a human operator touches a first location on the primary physical document and generally contemporaneously speaks into said microphone first location coordinates of the first location that is touched on the primary physical document, and after the human operator touches at least a second location on the primary physical document and generally contemporaneously speaks into said microphone second location coordinates of the second location that is touched on the primary physical document, said computer system, to which touch location information electronically produced by touching and spoken coordinate information produced by speaking the first location coordinates and the second location coordinates into said microphone are communicated, electronically captures the touch location information and the spoken coordinate information and correlates any other locations on the primary physical document with the positions on said touch board;
• a digital ink tablet cooperatively connected to said computer system, said digital ink tablet sized and shaped to accommodate the one or more secondary physical documents thereon such that when at least one of the one or more secondary physical documents is written or drawn on, secondary physical document writing is produced which is electronically captured and recognized by and communicated to said computer system;
• a speaker cooperatively connected to said computer system such that, when the one of the one or more secondary physical documents containing the secondary physical document writing is moved from said digital ink tablet and placed on the primary physical document positioned on said touch board, the location at which the one of the one or more of the secondary physical documents is placed on the primary physical document is electronically recognized by said computer system and said speaker broadcasts a request to which the human operator may communicate to said computer system information that identifies relative significance of the secondary physical document writing;
• said speaker, said microphone, and said computer system further cooperatively connected such that the human operator may communicate to said computer system, such as through said microphone, commands or inquiries which said computer system recognizes and analyzes and to which said computer system formulates and provides an output such as responses communicated audibly through said speaker;
• said computer system electronically combining the electronic inputs to said computer system, including the touch location information, the spoken coordinate information, the secondary physical document writing, significance identifying information, orally communicated information, and other multi-modal communications to form the integrated electronic version and from which an electronic display image is formed for displaying;
• a display cooperatively connected to said computer system such that the electronic display image is received from said computer system and displayed including for verification of some or all of the integrated electronic version;
• and
• a database cooperatively connected to said computer system for receipt and storage of the electronic inputs and the integrated electronic version that are transferable through a transfer system to one or more systems remote from said touch board that includes the primary physical document on which the one or more secondary physical documents are positioned and that thereby form the combination physical record that provides the backup.

A method for augmenting a paper-based work practice such that a combination physical record formed from a combination of a primary physical document on which one or more secondary physical documents are positioned is retained in hard copy form to provide a backup and an electronic duplicate of some or all of the combination physical record is prepared and made accessible for analysis and communication, said augmenting method comprising:

• orienting the primary physical document relative to a touch board cooperatively connected to a computer system, said orienting step comprising the steps of:
• placing the primary physical document on the touch board such that locations on the primary physical document are relative to positions on the touch board;
• touching a first location on the primary physical document to produce first touch location information and generally contemporaneously speaking into a microphone cooperatively connected to the computer system first location coordinates of the first location on the primary physical document that is touched to produce spoken first location coordinate information;
• touching at least a second location on the primary physical document to produce second touch location information and generally contemporaneously speaking into the microphone second location coordinates of the second location on the primary physical document that is touched to produce spoken second location coordinate information;
• and
• analyzing the first touch location information and the spoken first location coordinate information and the second touch location information and the spoken second location coordinate information by the computer system such that generally all other locations on the primary physical document are correlated with the positions on the touch board;
• positioning the secondary physical document on a digital ink tablet sized and shaped to accommodate and to permit writing or drawings made relative to the secondary physical document such that secondary physical document writing is produced which is electronically captured and recognized and communicated from the digital ink tablet computer to the computer system;
• writing or drawing relative to the secondary physical document to produce the secondary physical document writing;
• orally communicating oral inputs to the computer system;
• removing the secondary physical document from the digital ink tablet;
• placing the secondary physical document on the primary physical document;
• responding to a request from the computer system to identify relative significance of the secondary physical document writing such that the response is transmitted to the computer system;
• analyzing the secondary physical document writing, the placing of the secondary physical document on the primary physical document, and the oral inputs for meaning and forming an integrated version and an electronic projection image for projecting;
• projecting the electronic projection image so that the electronic projection image is comparable to the secondary physical document writing;
• and
• storing at least the integrated version and information at least with respect to the secondary physical document in a database cooperatively connected to the computer system for receipt and storage of and so that the information is accessible and transferrable through a transfer system to one or more systems remote from the touch board on which the primary physical document and one or more of the secondary physical documents are positioned and that thereby form the combination physical record that provides the backup.

A method for recording and displaying information such that a combination physical record formed from a combination of a primary physical document and one or more secondary physical documents is retained in hard copy form to provide a backup and at least an electronic duplicate of some or all of the combination physical record is prepared and made accessible for analysis and communication, said recording and displaying method comprising:

• positioning the primary physical document such that a machine vision system cooperatively connected to a computer system may be used to detect one or more writings or drawings made relative to the primary physical document;
• writing or drawing on the primary physical document to form primary physical document writing;
• additional writing or drawing on one or more secondary physical document to produce secondary physical document writing that may be electronically captured and recognized through use of the computer system;
• identifying through oral inputs to the computer system relative significance to form an electronic duplicate generally contemporaneously with said writing or drawing or said additional writing or drawing;
• electronically capturing and recognizing through the use of the computer system the primary physical document writing and the secondary physical document writing to form the electronic duplicate;
• placing the one or more secondary physical documents on the primary physical document such that the machine vision system cooperatively connected to the computer system may be used to detect identity or location of the one or more secondary physical documents;
• evaluating all or a portion of the primary physical document writing and the identity or location of the one or more secondary physical documents through use of the machine vision system;
• providing a response to a request from the computer system to identify additional relative significance of the secondary physical document writing such that the response is transmitted to the computer system;
• analyzing the oral inputs, the response, and other multi-modal communications for meaning;
• forming an electronic display image for displaying;
• displaying the electronic display image so that the electronic display image is comparable to the secondary physical document writing;
• and
• storing information at least with respect to the one or more secondary physical documents in a database cooperatively connected to the computer system for receipt and storage of and so that the information is accessible and transferrable as the electronic duplicate through a transfer system to one or more systems remote from the primary physical document and the one or more of the secondary physical documents and that thereby form the combination physical record that provides the backup.

A method for extending a paper-based work practice such that a primary physical document is retained in hard copy form to provide a backup and an electronic duplicate is prepared and made accessible for analysis and communication, said recording and displaying method comprising:

• positioning the primary physical document such that a machine vision system cooperatively connected to a computer system may be used to detect one or more writings or drawings made relative to a surface of the primary physical document;
• writing or drawing relative to the primary physical document to produce primary physical document writing that may be electronically detected, captured, and recognized through use of the machine vision system;
• identifying through oral inputs to the computer system relative significance of the primary physical document writing generally contemporaneously with said writing or drawing on the primary physical document;
• electronically detecting and capturing through the use of the machine vision system and recognizing through the use of the computer system the primary physical document writing on the primary physical document;
• detecting location of any one or all of the primary physical document writings that may be made on the surface of the primary physical document through use of the machine vision system;
• providing a response to a request from the computer system to identify additional relative significance of the primary physical document writing such that the response is transmitted to the computer system;
• analyzing the response for meaning;
• facilitating formation of an electronic display image for displaying;
• displaying the electronic display image so that the electronic display image is comparable to the primary physical document writing;
• and
• storing information received by the computer system in a database so that the information is accessible and transferable as the electronic duplicate through a transfer system to one or more systems remote from the primary physical document provided as the backup to the paper-based work practice.

A method for electronically extending a paper-based work practice such that a physical document that may be formed from a combination of one or more other documents is retained in hard copy form to provide a backup and an integrated electronic version, that includes electronic inputs including an electronic capture of some or all of the physical document is prepared and made accessible for analysis and communication, said electronically extending method comprising:

• positioning the primary physical document such that a machine vision system cooperatively connected to a computer system may be used to electronically capture one or more writings or drawings made relative to the primary physical document;
• writing or drawing relative to the primary physical document to produce primary physical document writing that may be electronically captured through use of the machine vision system;
• electronically capturing to the computer system through the use of the machine vision system the primary physical document writing;
• identifying through oral inputs to the computer system relative significance of the primary physical document writing, the oral inputs provided generally contemporaneously with said writing or drawing on the primary physical document;
• detecting at least location of any one or all of the primary physical document writings that may be made on the surface of the primary physical document through use of the machine vision system;
• providing a response to a request from the computer system to identify additional relative significance of the primary physical document writing such that the response is transmitted to the computer system;
• interacting through added oral inputs to the computer system including in answer to audible queries from the computer system;
• analyzing and electronically combining the electronically captured primary physical document writing, the oral inputs, the detected locations, the significance response, the added oral inputs, and other multi-modal communications that form some or all of the electronic inputs to provide the integrated electronic version from which an electronic display image may be produced;
• displaying the electronic display image so that the electronic display image may be compared and verified with some or all of the electronic inputs;
• and
• storing the electronic inputs and the integrated electronic version in a database so that the electronic inputs and the integrated electronic version is accessible and transferable through a transfer system such as to one or more systems remote from the primary physical document and the one or more other documents retained in the hard copy form to provide the backup.