# US 12,008,153 B2 · Interactive augmented reality experiences using positional tracking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,153 B2 - Document ID: US12008153B2 - Status: Granted - Kind: B2 - Application number: 17/988,526 - Filed: 2022-11-16 - Published: 2023-03-16 - Granted: 2024-06-11 - Earliest claimed: 2020-05-26 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Shin Hwun Kang; Ilteris Canberk; James Powderly; Dmytro Kucher; Dmytro Hovorov - Web page: https://patentgrep.com/patent/US12008153B2 ## CPC - A63F 13/26: having at least one additional display device, e.g. on the game controller or outside a game booth - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/42: by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle - A63F 13/46: Computing the game score - A63F 13/5375: for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game - A63F 13/577: using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars - A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition - G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)} - G02B 27/0172: {characterised by optical features} - G02B 27/0179: {Display position adjusting means not related to the information to be displayed} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06T 7/73: using feature-based methods - A63F 2300/8082: Virtual reality - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06T 2207/30208: Marker matrix - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Interactive augmented reality experiences with an eyewear device including a position detection system and a display system. The eyewear device registers a first marker position for a user-controlled virtual game piece and a second marker for an interaction virtual game piece. The eyewear device monitors its position (e.g., location and orientation) and updates the position of the user-controlled virtual game piece accordingly. The eyewear device additionally monitors the position of the user-controlled virtual game piece with respect to the interaction virtual game piece for use in generating a score. Augmented reality examples include a “spheroidal balancing” augmented reality experience and a “spheroidal balancing” augmented reality experience. ## Claims ### Claim 1 An interactive augmented reality system comprising: a position detection system; a display system; an eyewear device comprising the position detection system, the display system, a processor, and a memory; and programming in the memory, wherein execution of the programming by the processor configures the processor to perform functions, including functions to: register, with the position detection system, a marker position for a user-controlled virtual game piece, the marker position defined with respect to the eyewear device and responsive to movement of the eyewear device; present, via the display system, the user-controlled virtual game piece at the marker position; register, with the position detection system, another marker position for an interaction virtual game piece, the other marker position defined with respect to the eyewear device; present, via the display system, the interaction virtual game piece at the other marker position; update the marker position responsive to movement of the eyewear device; monitor interaction between the user-controlled virtual game piece and the interaction virtual game piece as the marker position updates; determine steps taken by a user of the eyewear device; and generate a score responsive to the steps taken by the user while the user-controlled virtual game piece is within a predefined distance of the interaction virtual game piece. ### Claim 2 (depends on 1) The system of claim 1, wherein the position detection system comprises: a camera coupled to the eyewear device, the camera configured to capture sequences of frames of video data, wherein each frame of video data comprises depth information for a plurality of pixels. ### Claim 3 (depends on 1) The system of claim 1, wherein the display system comprises: a see-through display supported by the eyewear device to present the user-controlled virtual game piece and the interaction virtual game piece. ### Claim 4 (depends on 1) The system of claim 1, wherein to determine the steps taken, programming in the memory further configures the processor to perform further functions, including a function to: increment a counter corresponding to the score for each step the user takes while the user-controlled virtual game piece is within the predefined distance of the interaction virtual game piece. ### Claim 5 (depends on 4) The system of claim 4, wherein the eyewear device further comprises an inertial measurement unit and wherein to determine the steps taken, programming in the memory further configures the processor to perform further functions, including functions to: monitor the inertial measurement unit; and increment the counter in response to measurements by the inertial measurement unit that are indicative of a step while the user-controlled virtual game piece is within the predefined distance of the interaction virtual game piece. ### Claim 6 (depends on 4) The system of claim 4, wherein to determine the steps taken, programming in the memory further configures the processor to perform further functions, including functions to: monitoring the position of the eyewear device; and incrementing the counter in response to a change in the monitored position exceeding a predefined step size while the user-controlled virtual game piece is within the predefined distance of the interaction virtual game piece. ### Claim 7 (depends on 1) The system of claim 1, further comprising: a physics engine; wherein execution of the programming by the processor further configures the eyewear device to: monitor the user-controlled virtual game piece; monitor the interaction virtual game piece; apply the physics engine at an interface between the user-controlled virtual game piece and the interaction virtual game piece to derive position updates for the other marker position; and update the other marker position with the derived position updates. ### Claim 8 (depends on 1) The system of claim 1, further comprising: a prediction engine; wherein execution of the programming by the processor further configures the eyewear device to: monitor, with the position detection system, a physical position of the eyewear device; predict, with the prediction engine, a heading of the eyewear device; and update the other marker position such that the interaction virtual game piece appears along the predicted heading. ### Claim 9 An interactive augmented reality method for use with an eyewear device having a position detection system and a display system, the method comprising: registering, with the position detection system, a marker position for a user-controlled virtual game piece, the marker position defined with respect to the eyewear device and responsive to movement of the eyewear device; presenting the user-controlled virtual game piece on the display system at the marker position; registering, with the position detection system, another marker position for an interaction virtual game piece, the other marker position defined with respect to the eyewear device; presenting the interaction virtual game piece on the display system at the other marker position; updating the marker position responsive to movement of the eyewear device; monitoring interaction between the user-controlled virtual game piece and the interaction virtual game piece as the marker position updates; determining steps taken by a user of the eyewear device; and generating a score responsive to the steps taken by the user while the user-controlled virtual game piece is within a predefined distance of the interaction virtual game piece. ### Claim 10 (depends on 9) The method of claim 9, wherein the display system includes a see-through display supported by the eyewear device and wherein the presenting steps comprise: presenting the user-controlled virtual game piece and the interaction virtual game piece on the see-through display. ### Claim 11 (depends on 9) The method of claim 9, wherein the determining comprises: incrementing a counter corresponding to the score for each step the user takes while the user-controlled virtual game piece is within the predefined distance of the interaction virtual game piece. ### Claim 12 (depends on 11) The method of claim 11, wherein the eyewear device further comprises an inertial measurement unit and wherein the determining the steps taken comprises: monitoring the inertial measurement unit; and incrementing the counter in response to measurements by the inertial measurement unit that are indicative of a step while the user-controlled virtual game piece is within the predefined distance of the interaction virtual game piece. ### Claim 13 (depends on 11) The method of claim 11, wherein the determining the steps taken comprises: monitoring the position of the eyewear device; and incrementing the counter in response to a change in the monitored position exceeding a predefined step size while the user-controlled virtual game piece is within the predefined distance of the interaction virtual game piece. ### Claim 14 (depends on 9) The method of claim 9, further comprising: updating the other marker position; wherein the monitoring interaction includes monitoring interaction between the user-controlled virtual game piece and the interaction virtual game piece as the marker position and the other marker position update. ### Claim 15 (depends on 14) The method of claim 14, wherein the eyewear device further has a physics engine and wherein the updating the other marker position comprises: monitoring the user-controlled virtual game piece; monitoring the interaction virtual game piece; applying the physics engine at an interface between the user-controlled virtual game piece and the interaction virtual game piece to derive position updates for the other marker position; and updating the other marker position with the derived position updates. ### Claim 16 (depends on 15) The method of claim 15, further comprising: identifying a starting position corresponding to the user-controlled virtual game piece while the interaction virtual game piece is in virtual contact with the user-controlled virtual game piece; monitoring a current position of the user-controlled virtual game piece while the interaction virtual game piece is in virtual contact with the user-controlled virtual game piece; and adjusting the score responsive to a distance between the identified starting position and the monitored current position. ### Claim 17 (depends on 15) The method of claim 15, wherein the user-controlled virtual game piece includes a planar surface and the interaction virtual game piece has a spheroidal surface, orientation of the planar surface is responsive to the position of the eyewear device, and the position of the spheroidal surface is responsive to the interface between the planar surface and the spheroidal surface. ### Claim 18 (depends on 9) The method of claim 9, wherein the eyewear device further has a prediction engine and wherein the updating the other marker position comprises: monitoring, with the position detection system, a physical position of the eyewear device; predicting, with the prediction engine, a heading of the eyewear device; and updating the other marker position such that the interaction virtual game piece appears along the predicted heading. ### Claim 19 A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of: registering, with a position detection system, a marker position for a user-controlled virtual game piece, the marker position defined with respect to an eyewear device and responsive to movement of the eyewear device; presenting the user-controlled virtual game piece on a display system at the marker position; registering, with the position detection system, another marker position for an interaction virtual game piece, the other marker position defined with respect to the eyewear device; presenting the interaction virtual game piece on the display system at the other marker position; updating the marker position responsive to movement of the eyewear device; monitoring interaction between the user-controlled virtual game piece and the interaction virtual game piece as the marker position updates; determining steps taken by a user of the eyewear device; and generating a score responsive to the steps taken by the user while the user-controlled virtual game piece is within a predefined distance of the interaction virtual game piece. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein to determine the steps taken, the program code, when executed, is operative to cause the electronic processor to perform the step of: incrementing a counter corresponding to the score for each step the user takes while the user-controlled virtual game piece is within the predefined distance of the interaction virtual game piece.