# US 2020/0396304 A1 · Synchronized console data and user interface playback

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0396304 A1
- Document ID: US20200396304A1
- Status: Published application
- Kind: A1
- Application number: 16/439,962
- Filed: 2019-06-13
- Published: 2020-12-17
- Earliest claimed: 2019-06-13
- Assignees: FullStory, Inc.
- Applicants: FullStory, Inc.
- Inventors: Joel Grayson Webber; Joshua Calvin Teague
- Web page: https://patentgrep.com/patent/US20200396304A1

## CPC

- H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)}
- G06F 8/38: for implementing user interfaces
- G06F 9/451: Execution arrangements for user interfaces
- G06F 11/302: {where the computing system component is a software system}
- G06F 11/327: {Alarm or error message display}
- G06F 11/3419: {by assessing time}
- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)}
- H04L 67/61: taking into account QoS or priority requirements
- H04L 67/75: Indicating network or usage conditions on the user display
- G06F 11/3041: {where the computing system component is an input/output interface (interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units G06F13/00)}
- G06F 2201/865: Monitoring of software
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing user interface development tools. In one aspect, a method includes receiving console logs that specify sequences of user interface events that occurred during user sessions. For each of multiple user sessions, a received console log for the user interface events that occurred during that user session is identified. Playback data that presents visual changes of one or more user interfaces of the publisher that occurred during that user session is generated. An interactive interface is generated and provided. The interface includes session replay area in which the playback data presents the visual changes of the one or more user interfaces of the publisher that occurred during that user session and a console log area that presents console entries from the console log for the user interface events that occurred during that user session.

## Claims

### Claim 1

A system comprising:
a data store; and
one or more computers that interact with the data store and execute instructions that cause the one or more computers to perform operations comprising:
receiving, from user devices of users, console logs that specify sequences of user interface events that occurred during user sessions in which the users interacted with user interfaces presented by one or more pages of a publisher, wherein the console logs include errors generated by the user devices during the user sessions;
for each user session of a plurality of different user sessions:
identifying a given console log for user interface events that occurred during the user session at a user device;
generating playback data that presents visual changes to one or more user interfaces of the publisher that occurred during the user session;
generating and providing an interactive interface that includes (i) session replay area in which the playback data presents the visual changes to the one or more user interfaces of the publisher that occurred during the user session and (ii) a console log area that presents console entries from the given console log, wherein the interactive interface includes an animation that visually specifies, during playback of the visual changes to the one or more user interfaces in the session replay area, which console entries in the given console log were generated by the user device as the visual changes occurred during the user session
updating the interactive interface to present page speed metrics for particular milestones in loading a given page presented during the user session;
detecting user interaction with a given page speed metric of the page speed metrics; and
generating a search for user sessions using search criteria based on the given page speed metric.

### Claim 2 (depends on 1)

The system of claim 1, wherein the animation comprises presenting a console log identifier element next to the console log entries in the given console log that were generated by the user device as the visual changes occurred during the user session.

### Claim 3 (depends on 1)

The system of claim 1, wherein the operations further comprise:
detecting user interaction with a given console log entry; and
causing the playback of the visual changes to the one or more user interfaces to present visual changes that occurred when the given console log entry was generated.

### Claim 4 (depends on 1)

The system of claim 1, wherein the interactive interface generated for the user session presents a list of network requests transmitted from the user device during the user session and, for each network request, a load time that represents a duration of time between a time at which the network request was transmitted and a time at which a user interface element requested by the network request was presented in one of the one or more user interfaces.

### Claim 5 (depends on 4)

The system of claim 4, wherein the list of network requests is visually updated during playback of the user session and based on the playback of the user session such that, when the playback of the user session presents a visualization of the user interface at a time at which a given network request was transmitted, the list of network requests includes the given network request.

### Claim 6 (depends on 4)

The system of claim 4, wherein the user devices generate network request data for each network request, the network request data including a user interface element requested by the network request and the load time for the network request.

### Claim 7 (depends on 6)

The system of claim 6, wherein generating the network request data by a given user device includes:
identifying, for a given network request, at least one of (i) a request message body that includes data being transmitted with the given network request and by the given user device or (ii) a received message body that includes data received by the given user device in response to the given network request;
examining each message body to identify portions of data having a data type that is not included in a whitelist of data types;
removing the identified portions of data from each message body; and
including, in the network request data, each remaining portion of data of each message body.

### Claim 8 (depends on 4)

The system of claim 4, wherein the operations further comprise:
detecting selection of a given network request in the list of network requests; and
visually updating the interactive interface to present a network request details panel that includes a timing summary for the given network request, the timing summary including measured time durations for events related to requesting and loading one or more user interface elements that were requested by the given network request.

### Claim 9 (depends on 4)

The system of claim 4, wherein the operations further comprise:
detecting user interaction with a network request in the list of network requests; and
causing the playback of the visual changes to the one or more user interfaces to present visual changes that occurred when the network request was transmitted.

### Claim 10 (depends on 1)

The system of claim 1, wherein the operations further comprise:
determining, for each page of a publisher, an average load time for the page;
determining, based on each determined load time, a set of slowest loading pages of the publisher; and
updating the interactive interface to present data identifying the set of slowest loading pages.

### Claim 11

(canceled)

### Claim 12 (depends on 1)

The system of claim 1, wherein the criteria specify a page speed metric that is greater than or equal to the given page speed metric.

### Claim 13 (depends on 1)

The system of claim 1, wherein the operations further comprise:
identifying a given page that was presented during the user session and had a load time that exceeds a threshold load time;
receiving, for each user interface element loaded in the given page during the user session, one or more first time durations that each represent a duration of time for the user device to request or load the user interface element and one or more second time durations that each represent a duration of time for the user interface element to be received by the user device in response to a request received from the user device; and
determining, based on the one or more first time durations and the one or more second time durations for each user interface element, whether the user device or a network across which the request was transmitted caused the load time for the given page to exceed the threshold load time.

### Claim 14

A method, comprising:
receiving, from user devices of users, console logs that specify sequences of user interface events that occurred during user sessions in which the users interacted with user interfaces presented by one or more pages of a publisher, wherein the console logs include errors generated by the user devices during the user sessions;
for each user session of a plurality of different user sessions:
identifying a given console log for user interface events that occurred during the user session at a user device;
generating playback data that presents visual changes to one or more user interfaces of the publisher that occurred during the user session;
generating and providing an interactive interface that includes (i) session replay area in which the playback data presents the visual changes to the one or more user interfaces of the publisher that occurred during the user session and (ii) a console log area that presents console entries from the given console log for the user interface events that occurred during the user session, wherein the interactive interface includes an animation that visually specifies, during playback of the visual changes to the one or more user interfaces in the session replay area, which console entries in the given console log were generated by the user device as the visual changes occurred during the user session;
updating the interactive interface to present page speed metrics for particular milestones in loading a given page presented during the user session;
detecting user interaction with a given page speed metric of the page speed metrics; and
generating a search for user sessions using search criteria based on the given page speed metric.

### Claim 15 (depends on 14)

The method of claim 14, wherein the animation comprises presenting a console log identifier element next to the console log entries in the given console log that were generated by the user device as the visual changes occurred during the user session.

### Claim 16 (depends on 14)

The method of claim 14, further comprising:
detecting user interaction with a given console log entry; and
causing the playback of the visual changes to the one or more user interfaces to present visual changes that occurred when the given console log entry was generated.

### Claim 17 (depends on 14)

The method of claim 14, wherein the interactive interface generated for the user session presents a list of network requests transmitted from the user device during the user session and, for each network request, a load time that represents a duration of time between a time at which the network request was transmitted and a time at which a user interface element requested by the network request was presented in one of the one or more user interfaces.

### Claim 18 (depends on 17)

The method of claim 17, wherein the list of network requests is visually updated during playback of the user session and based on the playback of the user session such that, when the playback of the user session presents a visualization of the user interface at a time at which a given network request was transmitted, the list of network requests includes the given network request.

### Claim 19 (depends on 17)

The method of claim 17, wherein the user devices generate network request data for each network request, the network request data including a user interface element requested by the network request and the load time for the network request.

### Claim 20

A non-transitory computer readable medium storing instructions that upon execution by one or more computers cause the one or more computers to perform operations comprising:
receiving, from user devices of users, console logs that specify sequences of user interface events that occurred during user sessions in which the users interacted with user interfaces presented by one or more pages of a publisher, wherein the console logs include errors generated by the user devices during the user sessions;
for each user session of a plurality of different user sessions:
identifying a given console log for user interface events that occurred during the user session at a user device;
generating playback data that presents visual changes to one or more user interfaces of the publisher that occurred during the user session;
generating and providing an interactive interface that includes (i) session replay area in which the playback data presents the visual changes to the one or more user interfaces of the publisher that occurred during the user session and (ii) a console log area that presents console entries from the given console log, wherein the interactive interface includes an animation that visually specifies, during playback of the visual changes to the one or more user interfaces in the session replay area, which console entries in the given console log were generated by the user device as the visual changes occurred during the user session;
updating the interactive interface to present page speed metrics for particular milestones in loading a given page presented during the user session;
detecting user interaction with a given page speed metric of the page speed metrics; and
generating a search for user sessions using search criteria based on the given page speed metric.

### Claim 21 (depends on 14)

The method of claim 14, wherein the search criteria specify a page speed metric that is greater than or equal to the given page speed metric.
