# US 8,745,090 B2 · System and method for exploring 3D scenes by pointing at a reference object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,745,090 B2 - Document ID: US8745090B2 - Status: Granted - Kind: B2 - Application number: 12/645,243 - Filed: 2009-12-22 - Published: 2010-12-02 - Granted: 2014-06-03 - Earliest claimed: 2008-12-22 - Assignees: Ipointer, Inc. - Applicants: David Caduff - Inventors: David Caduff - Web page: https://patentgrep.com/patent/US8745090B2 ## CPC - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - H04L 67/52: specially adapted for the location of the user terminal - H04W 4/02: Services making use of location information - H04W 4/026: {using orientation information, e.g. compass} - H04W 4/185: {by embedding added-value information into content, e.g. geo-tagging} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A system and method for enhancing location-based services by enabling spatial database systems to respond to or answer spatial queries that use a reference object to identify objects or features of interest in an environmental scene before a system user. The system and method for enhancing pointing technology by permitting system users to use queries to identify objects or features within the system user's field of view by pointing at the reference object or feature, and linking it to the object of interest by using spatial prepositions. ## Claims ### Claim 1 A computer-implemented method for responding to spatial queries about a real-world object of interest using a real-world reference object to identify the object of interest, comprising the steps of: (A) generating a query statement relating to at least one real-world object of interest with an electronic pointing device associated with a user, with the query statement including at least a query topic and a spatial preposition, and determining a geodetic position and a pointing direction of the pointing device, and transmitting the query statement, the geodetic position and the pointing direction to a system server; (B) mapping a projective three-dimensional representation of a real-world scene including at least the object of interest and the reference object, and transmitting the three-dimensional representation to the system server; (C) a system server performing substeps of, (1) generating a real-world two-dimensional digital representation of the three-dimensional scene mapped at step (B), with the two-dimensional digital representation including at least a digital representation of the object of interest and the reference object, (2) identifying the reference object and the reference object position according to the pointing device geodetic position and the pointing direction applied to the two-dimensional digital representation, wherein the pointing device geodetic position is according to a latitude, longitude, elevation, pitch, and yaw of the pointing device, (3) annotating the two-dimensional digital representation for identifying the object of interest according to the position of the reference object, the query topic, and the spatial preposition, (4) identifying the object of interest according to annotated two-dimensional digital representation at substep (C) (3) by linking the query topic and the spatial preposition of the query statement to a particular object in the two-dimensional digital representation, wherein the spatial preposition including an egocentric frame of reference expressing at least a point of view of the user according to the pointing device geodetic position, and (5) the system server communicating with the pointing device and transmitting at least an identification of the particular object as the object of interest to the pointing device. ### Claim 2 (depends on 1) The method as recited in claim 1, wherein the pointing device and system server communicate wired or wirelessly. ### Claim 3 (depends on 1) The method as recited in claim 1, wherein the query statement includes the object of interest as a query topic. ### Claim 4 (depends on 1) The method as recited in claim 1, wherein the spatial preposition includes a spatial relationship between the object of interest and the reference object. ### Claim 5 (depends on 4) The method as recited in claim 4, wherein the egocentric frame of reference identifies the object of interest according to the reference object position and the pointing device geodetic position. ### Claim 6 A computer-implemented method for responding to spatial queries about a real-world object of interest using a real-world reference object to identify the object of interest, comprising steps of: (A) generating a query statement relating to at least one real-world object of interest with an electronic pointing device associated with a user, with the query statement including at least a query topic and a spatial preposition, and determining a geodetic position and a pointing direction of the pointing device, and transmitting the query statement, the geodetic position and the pointing direction to a system server; (B) mapping a projective three-dimensional representation of a real-world scene including at least the object of interest and the reference object, and transmitting the three-dimensional representation to the system server; (C) a system server performing substeps of, (1) generating a real-world two-dimensional digital representation of the three-dimensional scene mapped at step (B), with the two-dimensional digital representation including at least a digital representation of the object of interest and the reference object, (2) identifying the reference object and the reference object position according to the pointing device geodetic position and the pointing direction applied to the two-dimensional digital representation, wherein the pointing device geodetic position is according to a latitude, longitude, elevation, pitch, and yaw of the pointing device, (3) identifying the object of interest according to the position of the reference object, the query topic and the spatial preposition by linking the query topic and the spatial preposition of the query statement to a particular object in the two-dimensional digital representation, wherein the spatial preposition including an egocentric frame of reference expressing at least a point of view of the user according to the pointing device geodetic position, and (4) the system server communicating with the pointing device and transmitting at least an identification of the particular object as the object of interest to the pointing device. ### Claim 7 (depends on 6) The method as recited in claim 6, wherein the pointing device and system server communicate wired or wirelessly. ### Claim 8 (depends on 6) The method as recited in claim 6, wherein the query statement includes the object of interest as a query topic. ### Claim 9 (depends on 6) The method as recited in claim 6, wherein the spatial preposition includes a spatial relationship between the object of interest and the reference object. ### Claim 10 (depends on 9) The method as recited in claim 9, wherein the spatial preposition further identifies the object of interest according to an absolute frame of reference expressing at least a geographically directional reference applied to the reference object position. ### Claim 11 (depends on 9) The method as recited in claim 9, wherein the spatial preposition further identifies the object of interest according to an intrinsic frame of reference expressing at least architecturally structural reference applied to the reference object position. ### Claim 12 A system for responding to spatial queries about a real-world object of interest using a real-world reference object to identify the object of interest, comprising: a pointing device associated with a user for pointing at a reference object in a real-world scene and generating a query statement including a query topic and a spatial preposition, relating to the object of interest according to a position of the reference object, determining a geodetic position and a pointing direction of the pointing device, and communicating the query statement, and the pointing device geodetic position and the pointing direction to a system server; and a system server further comprising, a mapping module for receiving and processing a projective three-dimensional representation of an environment that contains a real-world scene including the reference object and the object of interest, a scene generator that connects to the mapping module and receives an output from the mapping module for generating a real-world two-dimensional digital representation of the real-world scene including the reference object and the object of interest, and identifies the reference object according to the pointing device geodetic position and the pointing direction applied to the two-dimensional digital representation, wherein the pointing device geodetic position is according to a latitude, longitude, elevation, pitch, and yaw of the pointing device, a scene annotator for annotating the two-dimensional digital representation for identifying the object of interest according to the position of the reference object, the query topic and the spatial preposition of the query statement by linking the query topic and the spatial preposition to a particular object in the two-dimensional digital representation, wherein the spatial preposition including an egocentric frame of reference expressing a user's point of view according to the pointing device geodetic position, and communicating to the pointing device an identification of the particular object as the object of interest. ### Claim 13 (depends on 12) The system as recited in claim 12, wherein the pointing device includes a mobile device. ### Claim 14 (depends on 12) The system as recited in claim 12, wherein the egocentric frame of reference identifies the object of interest according to the reference object position and pointing device geodetic position. ### Claim 15 (depends on 12) The system as recited in claim 12, wherein the pointing device and system server communicate wired and wirelessly. ### Claim 16 A system for responding to spatial queries about a real-world object of interest using a reference object to identify the object of interest, comprising: a pointing device associated with a user for pointing at a reference object in a real-world scene and generating a query statement including a query topic and a spatial preposition, the query statement relating to the object of interest according to a geodetic position of the reference object, determining a geodetic position and a pointing direction of the pointing device, and communicating the query statement, and the pointing device geodetic position and the pointing direction to a system server; and a system server further comprising, a mapping module for receiving and processing a projective three-dimensional representation of an environment that contains a real-world scene including the reference object and the object of interest, a scene generator that connects to the mapping module and receives an output from the mapping module for generating a real-world two-dimensional digital representation of the real-world scene including the reference object and the object of interest, and identifies the reference object according to the pointing device geodetic position and the pointing direction applied to the two-dimensional digital representation, wherein the pointing device geodetic position is according to a latitude, longitude, elevation, pitch, and yaw of the pointing device, and an identifier module for identifying the object of interest in the two-dimensional digital representation according to the position of the reference object and the query topic and the spatial preposition of the query statement by linking the query topic and the spatial preposition to a particular object in the two-dimensional digital representation, and communicating to the pointing device an identification of the particular object as the object of interest wherein the spatial preposition including an egocentric frame of reference expressing a user's point of view according to the pointing device geodetic position. ### Claim 17 (depends on 16) The system as recited in claim 16, wherein the pointing device includes a mobile device. ### Claim 18 (depends on 16) The system as recited in claim 16, wherein the identifier module further includes applying an absolute frame of reference expressing at least a geographically directional reference for identifying the object of interest according to the reference object position. ### Claim 19 (depends on 16) The system as recited in claim 16, wherein the identifier module further includes applying an intrinsic frame of reference expressing at least an architecturally structural reference for identifying the object of interest according to the reference object position. ### Claim 20 (depends on 16) The system as recited in claim 16, wherein the pointing device and system server communicate wired and wirelessly.