# US 8,675,912 B2 · System and method for initiating actions and providing feedback by pointing at object of interest > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,675,912 B2 - Document ID: US8675912B2 - Status: Granted - Kind: B2 - Application number: 12/645,231 - Filed: 2009-12-22 - Published: 2010-12-02 - Granted: 2014-03-18 - Earliest claimed: 2008-12-22 - Assignees: Ipointer, Inc. - Applicants: David Caduff - Inventors: David Caduff - Web page: https://patentgrep.com/patent/US8675912B2 ## CPC - H04W 4/02: Services making use of location information - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - H04L 67/131: Protocols for games, networked simulations or virtual reality - H04W 4/026: {using orientation information, e.g. compass} - H04W 4/029: Location-based management or tracking services - H04W 4/185: {by embedding added-value information into content, e.g. geo-tagging} - H04L 67/52: specially adapted for the location of the user terminal - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A system and method as described for compiling feedback in command statements that relate to applications or services associated with spatial objects or features, pointing at such spatial object or feature order to identify the object of interest, and executing the command statements on a system server and attaching feedback information to their representation of this object or feature in a database of the system server. ## Claims ### Claim 1 A computer-implemented method for identifying an object in a real-world scene by pointing at the object with an electronic pointing device and electronically attaching feedback to the object in a digital representation of the object stored in system storage, comprising the steps of: (A) mapping a three-dimensional representation of the real-world scene containing at least one object and transmitting the three-dimensional representation to a system server; (B) inputting feedback relating to the at least one object to the pointing device; (C) inputting command statements relating to the at least one object to the pointing device; (D) generating a request at the pointing device that includes at least the feedback and/or command statements, and transmitting the request to the system server along with a pointing device geodetic position and pointing direction; and (E) the system server performing the Substeps of, (1) generating a two-dimensional digital representation of the three-dimensional scene mapped at Step (A), with the two-dimensional digital representation including at least a digital representation of the at least one object, (2) identifying the at least one object according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation, (3) processing the request and separating command statements and feedback from the request, (4) if the request includes only at least the command statement, executing the at least command statement and then performing Substep (E)(6), (5) if the request includes only feedback, linking the feedback to the digital representation of the at least one object stored in system storage and then performing Substep (E)(7), (6) if the request does not include at least one command statement than (E)(7), otherwise if the request includes at least one command statement and feedback, executing the at least one command statement and linking feedback to the digital representation of the at least one object stored in system storage and then performing Substep (E)(7), (7) the system server communicating to the pointing device execution of at least one command statement and/or the linking of feedback. ### Claim 2 (depends on 1) The method as recited in claim 1, wherein the pointing device and system server communicate wired or wirelessly. ### Claim 3 (depends on 1) The method as recited in claim 1, wherein the pointing device geodetic position includes a position according to a latitude, longitude, elevation, pitch, and yaw of the pointing device. ### Claim 4 (depends on 1) The method as recited in claim 1, wherein the request includes context information. ### Claim 5 (depends on 4) The method as recited in claim 4, wherein the method further includes adding context information to feedback. ### Claim 6 (depends on 5) The method as recited in claim 5, wherein the method further includes retrieving feedback relating to the at least one object when the at least one object is identified as an object of interest. ### Claim 7 (depends on 6) The method as recited in claim 6, wherein retrieving feedback includes retrieving feedback linked to the least one object and stored in system storage. ### Claim 8 (depends on 7) The method as recited in claim 7, wherein retrieving feedback includes retrieving context information. ### Claim 9 (depends on 1) The method as recited in claim 1, wherein the system storage stores environmental information for generating two-dimensional representations of real-world scenes and objects contained therein according to the pointing device geodetic position and pointing direction. ### Claim 10 (depends on 1) The method as recited in claim 1, wherein command statements include commands for affecting a real-world state of the at least one object. ### Claim 11 (depends on 1) The method as recited in claim 1, wherein linking feedback includes linking feedback to a subpart of the at least one object. ### Claim 12 (depends on 1) The method as recited in claim 1, the pointing device includes a mobile device. ### Claim 13 A computer-implemented method for identifying an object in a real-world scene by pointing at the object with an electronic pointing device and electronically attaching feedback information to the object in a digital representation of the object stored in system storage, comprising the steps of: (A) mapping a three-dimensional representation of the real-world scene containing at least one object and transmitting the three-dimensional representation to the system server; (B) inputting feedback relating to the at least one object to the pointing device; (C) generating a request at the pointing device that includes at least the feedback input at Step (B) and transmitting the request to the system server along with a geodetic pointing device position and pointing direction; and (D) the system server performing the Substeps of, (1) generating a two-dimensional digital representation of the three-dimensional scene mapped at Step (A), with the two-dimensional digital representation including at least a digital representation of the at least one object, (2) identifying the at least one object according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation, (3) linking the feedback to the digital representation of the at least one object stored in system server, and (4) the system server communicating to the pointing device the linking of feedback. ### Claim 14 (depends on 13) The method as recited in claim 13, wherein the pointing device and system server communicate wired or wirelessly. ### Claim 15 (depends on 13) The method as recited in claim 13, wherein the pointing device geodetic position includes a position according to a latitude, longitude, elevation, pitch, and yaw of the pointing device. ### Claim 16 (depends on 13) The method as recited in claim 13, wherein the request includes context information. ### Claim 17 (depends on 16) The method as recited in claim 16, wherein the method further includes adding context information to feedback. ### Claim 18 (depends on 17) The method as recited in claim 17, wherein the method further includes retrieving feedback relating to the at least one object when the at least one object is identified as an object of interest. ### Claim 19 (depends on 18) The method as recited in claim 18, wherein retrieving feedback includes retrieving at least feedback linked to the least one object and stored in the system storage. ### Claim 20 (depends on 19) The method as recited in claim 19, wherein retrieving feedback includes retrieving context information. ### Claim 21 (depends on 13) The method as recited in claim 13, wherein the system storage stores environmental information for generating two-dimensional representations of real-world scenes and objects contained therein according to the pointing device geodetic position and pointing direction. ### Claim 22 (depends on 13) The method as recited in claim 13, wherein linking feedback includes linking feedback to a subpart of the at least one object. ### Claim 23 (depends on 13) The method as recited in claim 13, the pointing device includes a mobile device. ### Claim 24 A system for identifying and attaching feedback to an object in a real-world scene, comprising, (A) a pointing device for pointing at an object of interest in a real-world scene and having input therein feedback and/or command statements with respect to the object of interest, with the pointing device determining a pointing device geodetic position and pointing direction, and generating a request including feedback and/or command statements, and communicating the request and pointing device geodetic position and pointing direction to a system server; (B) the system server further comprising, (1) a mapping module for receiving and processing a three-dimensional representation that contains the real-world scene including the object of interest, (2) a scene generator that connects to the mapping module and receives an output from the mapping module for generating a two-dimensional digital representation of the real-world scene including the object of interest and identifies the object of interest according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation, (3) a linker module for linking feedback from the request to the digital representation of the object of interest stored in system storage, (4) a command parser module for parsing command statements from the request, (5) a communications module for receiving requests, and the pointing device geodetic position and pointing direction from the pointing device, and transmitting at least retrieved feedback and/or command statement execution status to the pointing device; and (C) system storage in communication with the system server for storing feedback relating to objects, including the object of interest, and two-dimensional digital environment information relating to the real-world scene containing objects, including the object interest; and (D) an execution module in communications with the system server for executing command statements that affect a real-world state of the object of interest or subpart of the object of interest. ### Claim 25 (depends on 24) The system as recited in claim 24, wherein the pointing device includes a mobile device. ### Claim 26 (depends on 24) The system as recited in claim 24, wherein system storage includes system databases. ### Claim 27 (depends on 24) The system as recited in claim 24, wherein the execution module executes commands associated with appliances, applications, or services. ### Claim 28 (depends on 24) The system as recited in claim 24, wherein the system server and pointing device communicates wired or wirelessly.