# US 2024/0189724 A1 · Videogame telemetry data and game asset tracker for session recordings

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0189724 A1
- Document ID: US20240189724A1
- Status: Published application
- Kind: A1
- Application number: 18/508,016
- Filed: 2023-11-13
- Published: 2024-06-13
- Earliest claimed: 2020-03-30
- Assignees: Electronic Arts Inc.
- Applicants: Electronic Arts Inc.
- Inventors: Grace Yen
- Web page: https://patentgrep.com/patent/US20240189724A1

## CPC

- A63F 13/86: Watching games played by other players
- A63F 13/53: involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
- A63F 13/70: Game security or game management aspects
- A63F 2300/535: for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR

## Abstract

Disclosed are issue tracking systems, troubleshooting techniques, and user interfaces for troubleshooting, which are associated with a user of a software application triggering the recording of a session. During the session, the user may narrate suggestions or problems for the application while they interact with the application in real-time, and a recording engine of the application may record various types of session data, such as the user's interaction, narration, telemetry data, call stack data, and so forth. The session data is automatically submitted to an issue tracking system to process a support ticket. The issue tracking system may provide a user interface enabling a developer to review a support ticket and any associated session data to quickly determine the relevant portion of the application data (e.g., underlying program code) that needs to be modified. The issue tracking system may also process session data to identify related tickets or recurring issues.

## Claims

### Claim 1

A computer-implemented method performed by a user application on a user computing device, the method comprising:
receiving, within the user application, a user input to begin recording a session;
in response to receiving the user input, recording over a period of time:
video data of an interaction between the user application and a user, wherein the video data captures the user application being interacted with by the user;
audio data of a user narration during the interaction;
telemetry data generated by the user application during the interaction;
call stack data associated with execution of the user application during the user interaction; and
decoded frame data identifying visible elements of the user application within the video data of the interaction; and
transmitting, to an issue tracking system, the video data of the interaction, the audio data of the user narration, the telemetry data, the call stack data, and the decoded frame data.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
in response to receiving the user input, recording overlay data over the period of time, wherein the overlay data comprises user markups or annotations provided by the user during the interaction; and
transmitting, to the issue tracking system, the overlay data.

### Claim 3 (depends on 1)

The method of claim 1, wherein the video data of the interaction, the audio data of the narration, the telemetry data, the call stack data, and the decoded frame data are synchronized over the period of time.

### Claim 4 (depends on 1)

The method of claim 1, wherein the telemetry data comprises telemetry events.

### Claim 5 (depends on 1)

The method of claim 1, wherein the telemetry data, the call stack data, and the decoded frame data are synchronized over the period of time via timestamps.

### Claim 6 (depends on 1)

The method of claim 1, wherein the audio data of the user narration is from a webcam recording of the user.

### Claim 7 (depends on 1)

The method of claim 1, wherein the audio data of the user narration is from a microphone recording of the user.

### Claim 8 (depends on 1)

The method of claim 1, wherein the audio data of the user narration comprises an audio transcript.

### Claim 9 (depends on 1)

The method of claim 1, wherein the video data of the interaction comprises a plurality of frames, wherein the decoded frame data comprises a plurality of asset maps, and wherein each asset map of the plurality of asset maps corresponds to a frame of the plurality of frames.

### Claim 10 (depends on 1)

The method of claim 1, wherein the video data of the interaction comprises a plurality of frames, wherein the decoded frame data comprises a plurality of shader maps, and wherein each shader map of the plurality of shader maps corresponds to a frame of the plurality of frames.

### Claim 11 (depends on 1)

The method of claim 1, wherein the visible elements identified by the decoded frame data are models rendered by the user application.

### Claim 12 (depends on 1)

The method of claim 1, wherein the visible elements identified by the decoded frame data are pixels rendered by the user application.

### Claim 13

A computer-implemented method performed by an issue tracking system, the method comprising:
receiving one or more transmissions from a user application, wherein the one or more transmissions comprise:
video data of an interaction between the user application and a user, wherein the video data captures the user application being interacted with by the user;
overlay data comprising user markups or annotations provided by the user during the interaction;
audio data of a user narration during the interaction;
telemetry data generated by the user application during the interaction;
call stack data associated with execution of the user application during the interaction; and
decoded frame data identifying visible elements of the user application within the video data of the interaction,
wherein the video data of the interaction, the audio data of the user narration, the telemetry data, the call stack data, and the decoded frame data are synchronized over a period of time and are aggregately associated with a ticket; and
in response to receiving a request for the ticket from a developer, providing the video data of the interaction, the audio data of the user narration, the telemetry data, the call stack data, and the decoded frame data to the developer.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
generating data for a user interface, the user interface comprising:
a video view configured to display the video data of the interaction;
a user narration view configured to provide the audio data of the user narration, wherein the user narration view is synchronized to the video view; and
toggle-able views of the overlay data, the telemetry data, the call stack data, and the decoded frame data, wherein the toggle-able views are synchronized to the video view; and
providing the user interface for display to the developer.

### Claim 15 (depends on 13)

The method of claim 13, wherein the telemetry data comprises telemetry events.

### Claim 16 (depends on 13)

The method of claim 13, wherein the telemetry data, the call stack data, and the decoded frame data are synchronized over the period of time via timestamps.

### Claim 17 (depends on 13)

The method of claim 13, wherein the video data of the interaction comprises a plurality of frames, wherein the decoded frame data comprises a plurality of asset maps, wherein each asset map of the plurality of asset maps corresponds to a frame of the plurality of frames, and wherein the visible elements identified by the decoded frame data include models rendered by the user application.

### Claim 18

A system comprising:
a processor; and
a memory having instructions executable by the processor to cause the system to:
receive a user input to begin recording a session;
in response to receiving the user input, record over a period of time:
video data of the user application being interacted with by a user;
audio data of a user narration during the user application being interacted with by the user;
telemetry data generated by the user application during the user interaction;
call stack data associated with execution of the user application during the user application being interacted with by the user; and
decoded frame data identifying visible elements of the user application in the video data of the user application being interacted with by the user; and
transmit, to an issue tracking system, the video data, the audio data of the user narration, the telemetry data, the call stack data, and the decoded frame data.

### Claim 19 (depends on 18)

The system of claim 18, wherein the instructions are executable by the processor to further cause the system to:
in response to receiving the user input, record overlay data over the period of time, wherein the overlay data comprises user markups or annotations provided by the user during the user interaction with the user application; and
transmit, to the issue tracking system, the overlay data.

### Claim 20 (depends on 18)

The system of claim 18, wherein the telemetry data, the call stack data, and the decoded frame data are synchronized over the period of time via timestamps.
