# US 11,998,798 B2 · Virtual guided fitness routines for augmented reality experiences

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,998,798 B2
- Document ID: US11998798B2
- Status: Granted
- Kind: B2
- Application number: 17/321,178
- Filed: 2021-05-14
- Published: 2022-11-17
- Granted: 2024-06-04
- Earliest claimed: 2021-05-14
- Assignees: Snap Inc.
- Applicants: Megan Hong
- Inventors: Megan Hong
- Web page: https://patentgrep.com/patent/US11998798B2

## CPC

- A63B 24/0006: {Computerised comparison for qualitative assessment of motion sequences or the course of a movement}
- G16H 20/30: relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
- A61B 5/1122: {of movement trajectories}
- A61B 5/6803: {Head-worn items, e.g. helmets, masks, headphones or goggles}
- A61B 5/744: {Displaying an avatar, e.g. an animated cartoon character}
- A63B 24/0087: {Electric or electronic controls for exercising apparatus of groups A63B21/00 - A63B23/00, e.g. controlling load}
- A63B 71/0622: {Visual, audio or audio-visual systems for entertaining, instructing or motivating the user}
- G02B 27/0172: {characterised by optical features}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 1/169: {the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes (G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/03547: {Touch pads, in which fingers can move on a surface}
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G09B 21/006: {using audible presentation of the information}
- G16H 40/63: for local operation
- A63B 2024/0015: {Comparing movements or motion sequences with computerised simulations of movements or motion sequences, e.g. for generating an ideal template as reference to be achieved by the user}
- A63B 2024/0096: {using performance related parameters for controlling electronic or video games or avatars}
- A63B 2071/063: {Spoken or verbal instructions}
- A63B 2071/0638: {Displaying moving images of recorded environment, e.g. virtual environment}
- A63B 2071/0666: {worn on the head or face, e.g. combined with goggles or glasses}
- A63B 2071/068: {Input by voice recognition}
- A63B 2220/17: Counting, e.g. counting periodical movements, revolutions or cycles, or including further data processing to determine distances or speed
- A63B 2220/806: Video cameras
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/0141: {characterised by the informative content of the display}
- A63B: APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT (apparatus for passive exercising, massage A61H)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Example systems, devices, media, and methods are described for presenting a virtual guided fitness experience using the display of an eyewear device in augmented reality. A guided fitness application implements and controls the capturing of frames of motion data using an inertial measurement unit (IMU) and video data from one or more cameras. The method includes detecting exercise motions (with or without equipment) as well as detecting and counting repetitions. Relevant data about detected motions or equipment is retrieved and used to curate the guided fitness experience. A current rep count is presented on the display along with an avatar for playing messages, performing animated demonstrations, responding to commands and queries using speech recognition, and presenting guided fitness instructions through text, audio, and video.

## Claims

### Claim 1

A method of presenting a virtual fitness experience with an eyewear device, the eyewear device comprising a camera, an inertial measurement unit, a microphone, a loudspeaker, a guided fitness application, an image processing system, and a display, the method comprising:
capturing frames of motion data with the inertial measurement unit;
detecting a device motion of the eyewear device relative to a physical environment based on the captured frames of motion data;
determining whether the detected device motion matches a first predefined exercise activity from among a plurality of predefined exercise activities stored in an activity library, wherein each predefined exercise activity is characterized by an exercise data record comprising at least one of an identifier, one or more positions, or one or more motions;
retrieving a first exercise data record associated with the first predefined exercise activity; and
presenting on the display a virtual fitness experience based on the retrieved first exercise data record as an overlay relative to the physical environment.

### Claim 2 (depends on 1)

The method of claim 1, wherein the process of presenting a virtual fitness experience further comprises:
presenting an avatar at an avatar position relative to the display;
playing, through the avatar a start message associated with the retrieved first exercise data record;
playing, through the avatar a lecture associated with the retrieved first exercise data record, wherein the process of playing through the avatar comprises one or more operations selected from the group consisting of presenting text in a text bubble playing audio through the loudspeaker and presenting a video on the display; and
animating the avatar to perform a demonstration on the display adjacent the avatar position in correlation with a lesson wherein the demonstration is based on the retrieved first exercise data record.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
receiving a program selection;
retrieving program data based on the received program selection the program data comprising a program title and one or more session titles, each including an activity name, a recommended total repetitions, and a recommended duration;
presenting on the display at a session position the session title and a first activity name;
presenting a current recommended total repetitions with a repetitions window positioned on the display adjacent the session position;
presenting on the display the program title at a title position relative to the display;
playing, through the avatar a start message associated with the retrieved program data;
playing, through the avatar a lecture associated with the retrieved program data and
animating the avatar to perform a demonstration on the display adjacent the avatar position in correlation with a lesson wherein the demonstration is based on the retrieved program data.

### Claim 4 (depends on 3)

The method of claim 3, wherein the process of presenting a virtual fitness experience further comprises:
presenting on the display a graphical control element positioned adjacent the title position, the graphical control element comprising one or more controls selected from the group consisting of play, pause, back, next, and stop;
detecting a current fingertip location relative to a touchpad coupled to the eyewear device;
presenting a movable element at a current element position on the display in accordance with the detected current fingertip location;
identifying a first control of the graphical control element which is nearest to the current element position;
detecting a tapping gesture relative to the touchpad; and
executing a selecting action relative to the first control in accordance with the detected tapping gesture.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
detecting a subsequent device motion based on the captured frames of motion data;
retrieving a subsequent exercise data record associated with the detected subsequent device motion; and
presenting the virtual fitness experience based on the retrieved subsequent exercise data record.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
presenting on the display a current rep count with a repetition counter at an information position relative to the display, wherein the repetition counter is configured to selectively increment the current rep count and to begin with a start value that is correlated with the retrieved first exercise data record, which comprises a recommended total repetitions and a recommended duration;
detecting a repetitive motion of the eyewear device relative to the physical environment based on the captured frames of motion data;
determining whether the detected repetitive motion matches a first predefined repetition activity from among a plurality of predefined exercise activities stored in an activity library;
incrementing the current rep count by one in response to the first predefined repetition activity;
presenting a current time with a stopwatch positioned on the display adjacent the information position; and
incrementing the current time by one unit during the recommended duration.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
detecting a subsequent repetitive motion based on the captured frames of motion data;
determining whether the detected subsequent repetitive motion matches a subsequent predefined repetition activity; and
incrementing the current rep count by one in response to the subsequent predefined repetition activity.

### Claim 8 (depends on 1)

The method of claim 1, wherein the eyewear device further comprising a voice recognition module and wherein the method further comprises:
receiving human speech with the microphone;
converting the received speech into frames of audio data;
identifying, with the voice recognition module a first inquiry based on the frames of audio data; and
playing a first response through the loudspeaker in response to the identified first inquiry.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
capturing frames of video data within a field of view of the camera;
detecting, with the image processing system, an exercise apparatus within the captured frames of video data;
retrieving apparatus data associated with the detected exercise apparatus and
presenting the virtual fitness experience based on the retrieved apparatus data and the retrieved first exercise data record.

### Claim 10 (depends on 9)

The method of claim 9, wherein the process of presenting a virtual fitness experience further comprises:
presenting an avatar at an avatar position relative to the display;
playing, through the avatar a start message associated with the retrieved apparatus data;
playing, through the avatar a lecture associated with the retrieved apparatus data; and
animating the avatar to perform a demonstration on the display adjacent the avatar position in correlation with a lesson wherein the demonstration is based on the retrieved apparatus data.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
detecting, with the image processing system, a repetitive apparatus motion within the captured frames of video data;
determining whether the detected repetitive apparatus motion matches a first predefined apparatus repetition activity from among a plurality of predefined activities stored in an activity library; and
incrementing a current rep count by one in response to the first apparatus repetition activity.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
detecting a subsequent apparatus motion;
determining whether the subsequent apparatus motion matches a subsequent predefined apparatus repetition activity; and
incrementing the current rep count by one in response to the subsequent predefined apparatus repetition activity.

### Claim 13

A virtual guided fitness system, comprising:
an eyewear device comprising a camera, an inertial measurement unit, a microphone, a loudspeaker, a guided fitness application, an image processing system, and a display;
programming in a memory, wherein execution of the programming by a processor configures the eyewear device to perform functions, including functions to:
capture frames of motion data with the inertial measurement unit;
detect a device motion of the eyewear device relative to a physical environment based on the captured frames of motion data;
determine whether the detected device motion matches a first predefined exercise activity from among a plurality of predefined exercise activities stored in an activity library, wherein each predefined exercise activity is characterized by an exercise data record comprising at least one of an identifier, one or more positions, or one or more motions;
retrieve a first exercise data record associated with the first predefined exercise activity; and
present on the display a virtual fitness experience based on the retrieved first exercise data record as an overlay relative to the physical environment.

### Claim 14 (depends on 13)

The virtual guided fitness system of claim 13, wherein the function to present a virtual fitness experience further comprises functions to:
present an avatar at an avatar position relative to the display;
play, through the avatar a start message associated with the retrieved first exercise data record;
play, through the avatar a lecture associated with the retrieved exercise data wherein the process of playing through the avatar comprises one or more operations selected from the group consisting of presenting text in a text bubble playing audio through the loudspeaker and presenting a video on the display; and
animate the avatar using an avatar animation engine to perform a demonstration on the display adjacent the avatar position in correlation with a lesson wherein the demonstration is based on the retrieved first exercise data record.

### Claim 15 (depends on 13)

The virtual guided fitness system of claim 13, wherein execution of the programming by the processor further configures the eyewear device to perform additional functions, including functions to:
detect a subsequent device motion based on the captured frames of motion data;
retrieve a subsequent exercise data record associated with the detected subsequent device motion;
present the virtual fitness experience based on the retrieved subsequent exercise data record;
present on the display a current rep count with a repetition counter at an information position relative to the display, wherein the repetition counter is configured to selectively increment the current rep count and to begin with a start value that is correlated with the retrieved subsequent exercise data record, which comprises a recommended total repetitions and a recommended duration;
detect a repetitive motion of the eyewear device relative to the physical environment based on the captured frames of motion data;
determine whether the detected repetitive motion matches a first predefined repetition activity from among a plurality of predefined exercise activities stored in an activity library;
increment the current rep count by one in response to the first predefined repetition activity;
present a current time with a stopwatch positioned on the display adjacent the information position;
increment the current time by one unit during the recommended duration;
detect a subsequent repetitive motion based on the captured frames of motion data;
determine whether the detected subsequent repetitive motion matches a subsequent predefined repetition activity; and
increment the current rep count by one in response to the subsequent predefined repetition activity.

### Claim 16 (depends on 15)

The virtual guided fitness system of claim 15, wherein execution of the programming by the processor further configures the eyewear device to perform additional functions, including functions to:
detect, with the image processing system, a repetitive apparatus motion within the captured frames of video data;
determine whether the detected repetitive apparatus motion matches a first predefined apparatus repetition activity from among a plurality of predefined activities stored in an activity library;
increment the current rep count by one in response to the first apparatus repetition activity;
detect a subsequent apparatus motion;
determine whether the subsequent apparatus motion matches a subsequent predefined apparatus repetition activity; and
increment the current rep count by one in response to the subsequent predefined apparatus repetition activity.

### Claim 17

A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:
capturing frames of motion data with an inertial measurement unit of an eyewear device, the eyewear device further comprising a camera, a microphone, a loudspeaker, a guided fitness application, an image processing system, and a display;
detecting a device motion of the eyewear device relative to a physical environment based on the captured frames of motion data;
determining whether the detected device motion matches a first predefined exercise activity from among a plurality of predefined exercise activities stored in an activity library, wherein each predefined exercise activity is characterized by an exercise data record comprising at least one of an identifier, one or more positions, or one or more motions;
retrieving a first exercise data record associated with the first predefined exercise activity;
presenting on the display a virtual fitness experience based on the retrieved first exercise data record as an overlay relative to the physical environment wherein the step of presenting the virtual fitness experience further comprises:
presenting an avatar at an avatar position relative to the display;
playing, through the avatar a start message associated with the retrieved first exercise data record;
playing, through the avatar a lecture associated with the retrieved first exercise data record, wherein the process of playing through the avatar comprises one or more operations selected from the group consisting of presenting text in a text bubble playing audio through the loudspeaker and presenting a video on the display; and
animating the avatar to perform a demonstration on the display adjacent the avatar position in correlation with a lesson wherein the demonstration is based on the retrieved first exercise data record.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium storing program code of claim 17, wherein the step of presenting the virtual fitness experience further comprises:
presenting on the display a current rep count with a repetition counter at an information position relative to the display, wherein the repetition counter is configured to selectively increment the current rep count and to begin with a start value that is correlated with the retrieved first exercise data record, which comprises a recommended total repetitions and a recommended duration;
detecting a repetitive motion of the eyewear device relative to the physical environment based on the captured frames of the motion data;
determining whether the detected repetitive motion matches a first predefined repetition activity from among a plurality of predefined exercise activities stored in an activity library;
incrementing the current rep count by one in response to the first predefined repetition activity;
presenting a current time with a stopwatch positioned on the display adjacent the information position; and
incrementing the current time by one unit during the recommended duration.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium storing program code of claim 18, wherein the step of presenting the virtual fitness experience further comprises:
detecting, with the image processing system, a repetitive apparatus motion within the captured frames of video data;
determining whether the detected repetitive apparatus motion matches a first predefined apparatus repetition activity from among a plurality of predefined activities stored in an activity library; and
incrementing the current rep count by one in response to the first apparatus repetition activity.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium storing program code of claim 17, wherein the step of presenting the virtual fitness experience further comprises:
capturing frames of video data within a field of view of the camera;
detecting, with the image processing system, an exercise apparatus within the captured frames of video data;
retrieving apparatus data associated with the detected exercise apparatus; and
presenting the virtual fitness experience based on the retrieved apparatus data and the retrieved first exercise data record.