# US 8,625,852 B2 · System and method for linking real-world objects and object representations by pointing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,625,852 B2 - Document ID: US8625852B2 - Status: Granted - Kind: B2 - Application number: 13/947,719 - Filed: 2013-07-22 - Published: 2013-11-21 - Granted: 2014-01-07 - Earliest claimed: 2008-12-22 - Assignees: Ipointer, Inc. - Applicants: iPointer, Inc. - Inventors: David Caduff - Web page: https://patentgrep.com/patent/US8625852B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - H04W 4/185: {by embedding added-value information into content, e.g. geo-tagging} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A system and method are described for selecting and identifying a unique object or feature in the system user's three-dimensional (“3-D”) environment in a two-dimensional (“2-D”) virtual representation of the same object or feature in a virtual environment. The system and method may be incorporated in a mobile device that includes position and orientation sensors to determine the pointing device's position and pointing direction. The mobile device incorporating the present invention may be adapted for wireless communication with a computer-based system that represents static and dynamic objects and features that exist or are present in the system user's 3-D environment. The mobile device incorporating the present invention will also have the capability to process information regarding a system user's environment and calculating specific measures for pointing accuracy and reliability. ## Claims ### Claim 1 A computer-implemented method for determining a likelihood a moving object in a real-world scene is an object being pointed at by a mobile device that includes capabilities of a pointing device, comprising the steps of: (A) pointing a mobile electronic device that includes capabilities of a pointing device (mobile device) at a moving object of interest in the real-world scene; (B) generating with the mobile device a mobile device geodetic position and pointing direction, and transmitting the mobile device geodetic position and pointing direction, and standard deviations relating to generating the mobile device geodetic position and pointing direction to a system server; (C) mapping a three-dimensional representation of a real-world scene containing the objects in the real-world scene, including the moving object of interest, and transmitting the three-dimensional representation to the system server; (D) a system server performing the substeps of, (1) generating a two-dimensional virtual digital representation of the three-dimensional scene mapped at step (C), with the two-dimensional digital virtual representation including at least a virtual digital representation of the objects in the real-world scene, including the moving object of interest, (2) determining for each object, including the moving object of interest, in the two-dimensional virtual digital representation a distance estimate from the mobile device geodetic position to each such object, including the moving object of interest, (3) determining for each object, including the moving object of interest, in the two-dimensional virtual digital representation an accuracy of actual pointing and in determining the accuracy of pointing according to substep (D)(3) using standard deviations transmitted at step (B) caused by the mobile device in generating the mobile device geodetic position and pointing direction and the distance estimate for each object determined at substep (D)(2), (4) defining for each object, including the moving object of interest, in the two-dimensional virtual digital representation an optimal pointing location relating to a visible part of each object, including the moving object of interest, (5) determining for each object, including the moving object of interest, in the two-dimensional virtual digital representation a probability surface for optimal pointing from the mobile device geodetic position applying the definition for each object defined at substep (D)(4), (6) determining for each object, including the moving object of interest, in the two-dimensional virtual digital representation, a numeric correspondence between actual pointing accuracy and optimal pointing location by comparing a standard deviation for the actual pointing accuracy determined at substep (D)(3) and the probability surface for optimal pointing determined at substep (D)(5), (7) selecting as the object of interest the object with the highest numeric correspondence determined at substep (D)(6), (8) updating a position and direction of movement of each object, including the moving object of interest, in the two-dimensional virtual digital representation, and (9) the system server communicating to the mobile device an identity of the moving object of interest. ### Claim 2 (depends on 1) The method as recited in claim 1, wherein the mobile device and the system server communicate wired and wirelessly. ### Claim 3 (depends on 1) The method as recited in claim 1, wherein the mobile device geodetic position includes a position according to a latitude, longitude, elevation, pitch, and yaw of the mobile device. ### Claim 4 (depends on 1) The method as recited in claim 1, wherein the standard deviations relating to generating the mobile device geodetic position and pointing direction, include error propagation caused by mobile device sensors. ### Claim 5 (depends on 4) The method as recited in claim 4, wherein error propagations includes geodetic error propagation. ### Claim 6 (depends on 1) The method as recited in claim 1, wherein the distance estimate for each object, including the moving object of interest, includes a distance estimate from the mobile device geodetic position to a center of gravity of each object, including the moving object of interest. ### Claim 7 (depends on 1) The method as recited in claim 1, wherein a visible part of each object, including the moving object of interest, includes a visible part from the mobile device. ### Claim 8 (depends on 1) The method as recited in claim 1, wherein determining a probability surface for each object, including the moving object of interest, includes determining a probability surface according to an outline of each object, including the object of interest, from the mobile device geodetic position. ### Claim 9 (depends on 1) The method as recited in claim 1, wherein substep (D)(6) further includes determining a standard deviation for the actual pointing accuracy for each object, including the moving object of interest, in the two-dimensional virtual digital representation. ### Claim 10 (depends on 9) The method as recited in claim 9, wherein substep (D)(5) further includes determining for the probability surface for optional pointing a standard deviation for the optimal pointing location for each object, including the moving object of interest, in the two-dimensional virtual digital representation. ### Claim 11 (depends on 10) The method as recited in claim 10, wherein comparing at substep (D)(6) includes comparing the standard deviation for the actual pointing accuracy determined at substep (D)(3) and the standard deviation for the optimal pointing location using a t-test. ### Claim 12 (depends on 11) The method as recited in claim 11, wherein the numeric correspondence for each object, including the moving object of interest, includes a numeric correspondence of the overlap of the standard deviation for the actual pointing accuracy determined at substep (D)(3) and the standard deviation for the optimal pointing location. ### Claim 13 (depends on 12) The method as recited in claim 12, wherein numeric correspondence includes at least a lateral correspondence between the standard deviation for the actual pointing accuracy determined at substep (D)(3) and the standard deviation for optimal pointing location according to an x-axis in the two-dimensional virtual digital representation including the optimal point location and the actual pointing location. ### Claim 14 (depends on 12) The method as recited in claim 12, wherein numeric correspondence includes at least a vertical correspondence between the standard deviation for the actual pointing accuracy determined at substep (D)(3) and the standard deviation for the optimal pointing location according to a y-axis in the two-dimensional virtual digital representation including the optimal point location and the actual pointing location. ### Claim 15 (depends on 1) The method as recited in claim 1, wherein the mobile device includes a handheld device that includes capabilities of a pointing device. ### Claim 16 A system for determining a likelihood a moving object in a real-world scene is an object being pointed at by a mobile device that includes capabilities of a pointing device, comprising: a mobile device that includes the capabilities of a pointing device for generating a geodetic position and pointing direction for the mobile device, and transmitting the mobile device geodetic position and pointing direction, and standard deviations relating to generating the mobile device geodetic position and pointing direction to a system server; and a system server further comprising, a mapping module for receiving and processing a three-dimensional representation of an environment that contains a real-world scene with objects, including the moving object of interest, a scene generator module that connects to the mapping module and receives an output from the mapping module that generates a two-dimensional virtual digital representation of the real-world scene including objects, including the moving object of interest, and determines for each object, including the moving object of interest, in the two-dimensional virtual digital representation a distance estimate from the mobile device geodetic position to each such moving object, including the moving object of interest, a pointing accuracy module that determines for each object, including the moving object of interest, in the two-dimensional virtual digital representation an accuracy of actual pointing and in determining accuracy of pointing including the standard deviations transmitted from the mobile device caused by the mobile device in generating the mobile device geodetic position and pointing direction, and the distance estimate, optimal pointing module that defines for each object, including the moving object interest, in the two-dimensional virtual digital representation an optimal pointing location relating to a visible part of each object, including the moving object of interest, and for each object, including the moving object of interest, in the two-dimensional virtual digital representation a probability surface for optimal pointing from the mobile device geodetic position to such objects, including the moving object of interest, an updater for updating the position and/or direction of movement of each object of interest, including the moving object of interest, in the two-dimensional virtual digital representation in real time, and a comparison module that determines for each object, including the moving object of interest, in the two-dimensional virtual digital representation, a numeric correspondence between actual pointing accuracy and optimal pointing location by comparing a standard deviation for actual pointing accuracy and a probability surface for optimal pointing for each such object, including the moving object of interest, and that selects as the object of interest the object with the highest numeric correspondence according to a comparison by the comparison module, and a communications module that communicates with the mobile device with the communications including transmitting an identity of the moving object of interest to the mobile device. ### Claim 17 (depends on 15) The system as recited in claim 15, wherein the mobile device and system server communicate wired or wirelessly. ### Claim 18 (depends on 15) The system as recited in claim 15, wherein the mobile device includes sensors for determining mobile device geodetic location and pointing direction. ### Claim 19 (depends on 18) The system as recited in claim 18, wherein pointing device sensors cause mobile device geodetic position errors. ### Claim 20 (depends on 19) The system as recited in claim 19, when mobile device geodetic position includes a position according to a latitude, longitude, elevation, pitch, and yaw of the pointing device. ### Claim 21 (depends on 15) The system as recited in claim 15, wherein the comparison module includes a testing module for testing comparisons using a t-test. ### Claim 22 (depends on 16) The system as recited in claim 16, wherein the mobile device includes a handheld device that includes capabilities of a pointing device.