# US 12,602,880 B2 · Controlling augmented reality content via selection of real-world locations or objects > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,880 B2 - Document ID: US12602880B2 - Status: Granted - Kind: B2 - Application number: 18/083,344 - Filed: 2022-12-16 - Published: 2023-06-22 - Granted: 2026-04-14 - Earliest claimed: 2021-12-16 - Assignees: Niantic Spatial, Inc. - Applicants: Niantic Spatial, Inc. - Inventors: Christopher Grant Stoski; Marcus Romell Reed; Jose Juan Ceballos; Marcos Enrique Borregales; Justin Ryan Pando, I - Web page: https://patentgrep.com/patent/US12602880B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 15/06: Ray-tracing - G06T 19/003: {Navigation within 3D models or images} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A client device and a controller allow a user to control AR content by selecting real-world locations or objects. The client receives position data indicating a position and orientation of the controller, the position data defining an axis of the controller. The client device performs ray casting to determine a location in a 3D map of a real world that intersects the axis. The client device receives a selection indication (e.g., a user pressing a button on the controller). The client device selects, subsequent to the selection indication, the location in the 3D map that intersects the axis as a waypoint. The client device defines a route for a virtual object based on the waypoint. ## Claims ### Claim 1 A method for defining a route for a virtual object to travel along in an augmented reality (AR) application, the method comprising: receiving position data indicating a position and orientation of a controller, the position data defining an axis of the controller; performing ray casting to determine a location in a 3D map of a real world that intersects the axis; receiving a selection indication; selecting, subsequent to the selection indication, the location in the 3D map that intersects the axis as a waypoint; defining the route for the virtual object to travel along based on the waypoint; and providing, to a client device, AR data including the virtual object traversing along the route, wherein a movement speed of the virtual object along the route is based on a material of a real-world object along the route. ### Claim 2 (depends on 1) The method of claim 1, further comprising identifying the real-world object in the 3D map, wherein defining the route is further based on the real-world object. ### Claim 3 (depends on 1) The method of claim 1, further comprising identifying the material of the real-world object, wherein defining the route is further based on the identified material. ### Claim 4 (depends on 1) The method of claim 1, further comprising receiving traversal characteristics of the virtual object that will traverse the route, wherein defining the route is further based on the traversal characteristics of the virtual object. ### Claim 5 (depends on 1) The method of claim 1, wherein the waypoint defines an end point of the route. ### Claim 6 (depends on 5) The method of claim 5, wherein a start point of the route is based on a location of the virtual object in the 3D map. ### Claim 7 (depends on 1) The method of claim 1, wherein the waypoint defines a start point of the route. ### Claim 8 (depends on 1) The method of claim 1, further comprising providing, to the client device, AR data including the virtual object interacting with a real-world object at the waypoint. ### Claim 9 (depends on 1) The method of claim 1, wherein multiple waypoints are selected subsequent to receiving multiple selection indications, wherein the route is defined based on the multiple waypoints. ### Claim 10 (depends on 9) The method of claim 9, wherein the route is defined to intersect each of the multiple waypoints. ### Claim 11 (depends on 1) The method of claim 1, wherein the real-world object is a portion of ground and the material is a material of the portion of the ground. ### Claim 12 A non-transitory computer-readable medium storing instructions that, when executed by a computing system, cause the computing system to perform operations comprising: receiving position data indicating a position and orientation of a controller, the position data defining an axis of the controller; performing ray casting to determine a location in a 3D map of a real world that intersects the axis; receiving a selection indication; selecting, subsequent to the selection indication, the location in the 3D map that intersects the axis as a waypoint; defining a route for a virtual object to travel along based on the waypoint; and providing, to a client device, AR data including the virtual object traversing along the route, wherein a movement speed of the virtual object along the route is based on a material of a real-world object along the route. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the operations further comprise identifying the real-world object in the 3D map, wherein defining the route is further based on the real-world object. ### Claim 14 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the operations further comprise identifying the material of the real-world object, wherein defining the route is further based on the identified material. ### Claim 15 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the operations further comprise receiving traversal characteristics of the virtual object that will traverse the route, wherein defining the route is further based on the traversal characteristics of the virtual object. ### Claim 16 (depends on 12) The non-transitory computer-readable medium of claim 12, further comprising providing, to the client device, AR data including the virtual object interacting with a real-world object at the waypoint. ### Claim 17 A method for defining a route for a virtual object to travel along in an augmented reality (AR) application, the method comprising: receiving position data indicating a position and orientation of a controller, the position data defining an axis of the controller; performing ray casting to determine an object in a 3D map of a real world that intersects the axis; receiving a selection indication; selecting, responsive to the selection indication, the object in the 3D map that intersects the axis as a waypoint; defining the route to travel along based on the waypoint; and providing, to a client device, AR data including the virtual object traversing along the route, wherein a movement speed of the virtual object along the route is based on a material of a real-world object along the route. ### Claim 18 (depends on 17) The method of claim 17, further comprising identifying the real-world object in the 3D map, wherein defining the route is further based on the real-world object. ### Claim 19 (depends on 17) The method of claim 17, further comprising identifying the material of the real-world object, wherein defining the route is further based on the identified material. ### Claim 20 (depends on 17) The method of claim 17, further comprising receiving traversal characteristics of the virtual object that will traverse the route, wherein defining the route is further based on the traversal characteristics of the virtual object.