# US 7,466,436 B2 · Method of instructing a computer system using an interface surface > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 7,466,436 B2 - Document ID: US7466436B2 - Status: Granted - Kind: B2 - Application number: 11/026,045 - Filed: 2005-01-03 - Published: 2005-05-26 - Granted: 2008-12-16 - Earliest claimed: 1999-09-17 - Assignees: Silverbrook Research Pty Ltd - Inventors: Kia Silverbrook; Paul Lapstun - Web page: https://patentgrep.com/patent/US7466436B2 ## CPC - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06F 3/0321: {by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet (constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037)} - G06F 3/03545: {Pens or stylus} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 40/171: by use of digital ink - G06F 40/174: Form filling; Merging - G06Q 10/10: Office automation; Time management - H04N 1/00127: {Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture} - G06F 2203/0337: Status LEDs integrated in the mouse to provide visual feedback to the user about the status of the input device, the PC, or the user - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method of enabling instruction of a computer to perform tasks, using an interface surface having command and object interactive elements, the interface surface having command and object interactive elements, the interface surface including coded data. The method includes having the computer system receive indicating data from a sensing device, which is generated when the sensing device is moved relative to the interface surface, the indicating data being indicative of a stroke designating the interactive elements. The computer system uses the indicating data to determine the interactive elements designated by the stroke before applying a command to at least one object using the designated interactive elements. ## Claims ### Claim 1 A method for enabling instruction of a computer to perform tasks, the method including the steps of in a computer system: providing a user with a first interface surface having first interactive elements including at least one command interactive element and at least one object interactive element, the first interface surface including first coded data at least partially indicative of the interactive elements, wherein the first coded data is substantially invisible in the visible spectrum; receiving indicating data from a sensing device, the sensing device sensing at least some of the first coded data when the sensing device is moved relative to the first interface surface and generating the indicating data using the sensed first coded data, the indicating data being at least partially indicative of a stroke designating the first interactive elements, wherein the sensing device includes a marking nib; a non-marking nib; and a force sensor which indicates whether the sensing device is in contact with the first interface surface, wherein the indicating data received by the processing system is indicative of whether or not the sensing device is in contact with the first interface surface; determining, using the indicating data, the first interactive elements designated by the stroke; determining, using data stored therein, whether the sensing device is being operated using one of a marking nib and a non-marking nib; applying a command to at least one object using the designated interactive elements;- and in response to applying the command to the at least one object, and in response to determining that the sensing device is being operated using the non-marking nib, and without further user intervention, printing, using a single printer, a second interface surface, upon receiving the indicating data indicative of the sensing device failing to be in contact with the first interface surface, the interface surface having visual information representing a secondary interactive element and second coded data at least partially indicative of the secondary interactive element, wherein the second coded data is substantially invisible in the visible spectrum. ### Claim 2 (depends on 1) The method according to claim 1, wherein the method includes determining if the interactive elements that the stroke designates includes at least one command interactive element and at least one object interactive element, and on successful determination, applying the at least one associated command to the at least one associated object. ### Claim 3 (depends on 1) The method according to claim 1, wherein the method includes receiving indicating data is further indicative of a sequence of positions of the stroke, and wherein the method includes determining from the sequence of positions, at least one position being indicative of the at least one object interactive element, and at least one position being indicative of the at least one command interactive element, to thereby determine the designated interactive elements. ### Claim 4 (depends on 3) The method according to claim 3, wherein the stroke at least partially intersects at least one of the first interactive elements, wherein the method includes: determining, from the indicating data, at least one position coincident with an interactive element; and, identifying, using the determined position, the interactive element. ### Claim 5 (depends on 3) The method according to claim 3, wherein the stroke at least partially encircles at least one of the first interactive elements, wherein the method includes: determining from the indicating data, at least one encirclement including at least one interactive element; and, identifying, using the encirclement, at least one interactive element. ### Claim 6 (depends on 1) The method according to claim 1, wherein the stroke intersects a plurality of regions, at least one region being associated with an interactive element and wherein the method includes: determining, using the indicating data, the intersected regions; and, determining, using the determined intersected regions, at least one interactive element. ### Claim 7 (depends on 1) The method of claim 1, wherein at least one of the first coded data and the second coded data is indicative of an identity of at least one of: an interactive element; and, a region associated with an interactive element; wherein the method includes: determining, using the indicating data, an identity; and, determining, using the identity, the interactive element. ### Claim 8 (depends on 7) The method of claim 7, wherein the method includes: determining, using the identity, a page description, the page description being indicative of a layout of the coded data and the first interactive elements; and, determining, from the page description and the designated interactive elements, the task to be performed by the computer. ### Claim 9 (depends on 1) The method according to claim 1, the indicating data includes an identity of the first interface surface and a sequence of positions of the stroke and wherein the method includes: determining, using the identity of the surface, a page description, the page description being indicative of a layout of the first coded data and the first interactive elements; and, determining, using the sequence of positions of the stroke and the page description, the designated interactive element. ### Claim 10 A method for enabling instruction of a computer to perform tasks, the method including the steps of in a sensing device: sensing at least some first coded data provided on a first interface surface having first interactive elements including at least one command interactive element and at least one object indicating element, the first coded data being at least partially indicative of the interactive elements, wherein the first coded data is substantially invisible in the visible spectrum, wherein the sensing device includes: a marking nib; a non marking nib; and a force sensor which indicates whether the sensing device is in contact with the first interface surface; generating indicating data using the sensed first coded data, the indicating data being at least partially indicative of a stroke designating the first interactive elements and the force sensed by the force sensor to indicate whether the sensing device is in contact with the first interface surface; and, transferring the indicating data to a computer system, the computer system being responsive to: determining, using the indicating data, the first interactive elements designated by the stroke; applying a command to at least one object using the designated interactive elements; determining, in the computer system using data stored therein, whether the sensing device is being operated using one of a marking nib and a non-marking nib; and in response to applying the command to the at least one object, and in response to determining that the sensing device is being operated using the non-marking nib and without further user intervention, printing, using a single printer, a second interface surface, upon receiving the indicating data indicative of the sensing device failing to be in contact with the first interface surface, the interface surface having visual information representing a secondary interactive element and second coded data at least partially indicative of the secondary interactive element, wherein the second coded data is substantially invisible in the visible spectrum. ### Claim 11 (depends on 10) The method according to claim 10, wherein the first coded data is indicative of a position on the first interface surface, wherein the method includes: determining, using the sensed first coded data, a sequence of positions of the stroke; and, generating indicating data further indicative of the sequence of positions of the stroke. ### Claim 12 (depends on 11) The method according to claim 11, wherein the sequence of positions are time stamped and wherein the method includes generating indicating data further indicative of a sequence of time stamped positions of the stroke. ### Claim 13 (depends on 11) The method according to claim 11, wherein at least some of the sensed first coded data is indicative of the identity of the first interface surface, wherein the method includes: determining, using the sensed first coded data, the identity of the first interface surface; and, generating indicating data further indicative of the identity of the first interface surface. ### Claim 14 (depends on 10) The method according to claim 10, wherein the sensed first coded data is indicative of an identity of at least one region associated with the first interactive elements, wherein the method includes: determining, using the sensed first coded data, an identity for each sensed region and, generating indicating data further indicative of the identity of the region. ### Claim 15 (depends on 1) The method according to claim 1, wherein the method includes printing, using the single printer, the second coded data and the secondary interactive element such that at least a portion of the second coded data and at least a portion of the secondary interactive element are coincident on the secondary interface. ### Claim 16 (depends on 1) The method according to claim 1, wherein the sensing device includes a unique sensing identity, wherein the method includes: receiving data indicative of the unique sensing identity; and determining, using a record of registered sensing devices and based on the unique sensing identity, whether the second interface surface is allowed to be printed using the printer. ### Claim 17 (depends on 10) The method according to claim 10, wherein the method includes printing, using the single printer, the second coded data and the secondary interactive element such that at least a portion of the second coded data and at least a portion of the secondary interactive element are coincident on the secondary interface. ### Claim 18 (depends on 10) The method according to claim 10, wherein the sensing device includes a unique sensing identity, wherein the method includes: receiving data indicative of the unique sensing identity; and determining, using a record of registered sensing devices and based on the unique sensing identity, whether the second interface surface is allowed to be printed using the printer.