# US 12,182,594 B2 · Capturing and processing interactions with a user interface of a native application > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,594 B2 - Document ID: US12182594B2 - Status: Granted - Kind: B2 - Application number: 18/103,924 - Filed: 2023-01-31 - Published: 2023-08-10 - Granted: 2024-12-31 - Earliest claimed: 2017-07-18 - Assignees: FullStory, Inc. - Applicants: FullStory, Inc. - Inventors: Matthew Mastracci; Joel Grayson Webber; Michael Morrissey; Hollis Bruce Johnson, Jr. - Web page: https://patentgrep.com/patent/US12182594B2 ## CPC - G06F 9/451: Execution arrangements for user interfaces - G06F 3/023: Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06F 3/048: Interaction techniques based on graphical user interfaces [GUI] - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface of an application are disclosed. In one aspect, a method includes, for each of a plurality of different user sessions of a native application, accessing frame bundles that each include data representing content presented by a frame of a user interface of the native application at a given time. Each frame bundle includes at least a portion of a view tree of the native application used to generate the user interface at the given time and data specifying content presented by each view of the portion of the view tree. Based on the frame bundles, playback data are generated that present visual changes of the user interface corresponding to changes to the view trees. ## Claims ### Claim 1 A method comprising: initiating, at a client device, a user session with one or more user interfaces of an application running on the client device; initiating, at the client device, an instrumentation module for collecting event data for use in recreating playback of the user session at a computing device different from the client device; identifying a set of drawing functions of the application; modifying at least a subset of the drawing functions to send drawing data related to objects being drawn to a display of the client device to the instrumentation module, wherein the drawing data for at least one drawing function includes data that enables another computing system to redraw the object using data indicating the drawing function; for each frame of multiple frames of the user interface displayed during the user session: obtaining, by the instrumentation module, data representing a plurality of views drawn to a display of the client device to generate display of the frame; and obtaining, by the instrumentation module and for each view of one or more of the plurality of views, data specifying one or more drawing operations processed by the application to draw content of the view on the display of the client device, wherein, for views that include a modified drawing function of the subset of drawing functions, the data includes the drawing data received from the modified drawing function; generating the event data comprising, for each one or more of the multiple frames, the data representing plurality of views and the data specifying the one or more drawing operations; and sending the event data to the computing device. ### Claim 2 (depends on 1) The method of claim 1, wherein modifying the subset of drawing functions comprises replacing a drawing function with an instrumentation logging method. ### Claim 3 (depends on 1) The method of claim 1, wherein modifying the subset of drawing functions comprises modifying a drawing function using a reflection technique to capture drawing code of the drawing function. ### Claim 4 (depends on 1) The method of claim 1, wherein modifying the subset of drawing functions comprises adding a shim to intercept calls to the drawing function. ### Claim 5 (depends on 4) The method of claim 4, wherein the shim is configured to handle the calls to the drawing function or redirect the calls to the drawing function to another function. ### Claim 6 (depends on 1) The method of claim 1, wherein at least one drawing function comprises a drawing method. ### Claim 7 (depends on 1) The method of claim 1, wherein modifying the subset of drawing functions comprises associating image data with a resource name. ### Claim 8 (depends on 1) The method of claim 1, wherein modifying the subset of drawing functions comprises replacing a drawing method with a stub configured to perform functionality of the drawing method. ### Claim 9 (depends on 1) The method of claim 1, wherein modifying the subset of drawing functions comprises: detecting a drawing function configured to render bits; and inserting a shim that enables a canvas recorder class to call another drawing function. ### Claim 10 (depends on 1) The method of claim 1, wherein modifying the subset of drawing functions comprises augmenting a call to a drawing function with a call to an equivalent logging method. ### Claim 11 (depends on 1) The method of claim 1, further comprising identifying, for inclusion in the subset of the set of drawing functions, each occurrence of a drawing function in the application that is in a list of drawing functions into which a canvas recorder subclass is eligible to be passed. ### Claim 12 (depends on 1) The method of claim 1, wherein modifying the subset of drawing functions comprises instrumenting, by the instrumentation module, one or more drawing functions using one or more reflection techniques such that calls of each drawing function of the one or more drawing functions are replaced or augmented with a stub method that collects data related to calls of the drawing function. ### Claim 13 A system, comprising: one or more processors of a client device; and one or more memories having stored thereon computer readable instructions configured to cause the one or more processors to perform operations comprising: initiating, at the client device, a user session with one or more user interfaces of an application running on the client device; initiating, at the client device, an instrumentation module for collecting event data for use in recreating playback of the user session at a computing device different from the client device; identifying a set of drawing functions of the application; modifying at least a subset of the drawing functions to send drawing data related to objects being drawn to a display of the client device to the instrumentation module, wherein the drawing data for at least one drawing function includes data that enables another computing system to redraw the object using data indicating the drawing function; for each frame of multiple frames of the user interface displayed during the user session: obtaining, by the instrumentation module, data representing a plurality of views drawn to a display of the client device to generate display of the frame; and obtaining, by the instrumentation module and for each view of one or more of the plurality of views, data specifying one or more drawing operations processed by the application to draw content of the view on the display of the client device, wherein, for views that include a modified drawing function of the subset of drawing functions, the data includes the drawing data received from the modified drawing function; generating the event data comprising, for each one or more of the multiple frames, the data representing the plurality of views and the data specifying the one or more drawing operations; and sending the event data to the computing device. ### Claim 14 (depends on 13) The system of claim 13, wherein modifying the subset of drawing functions comprises replacing a drawing function with an instrumentation logging method. ### Claim 15 (depends on 13) The system of claim 13, wherein modifying the subset of drawing functions comprises modifying a drawing function using a reflection technique to capture drawing code of the drawing function. ### Claim 16 (depends on 13) The system of claim 13, wherein modifying the subset of drawing functions comprises adding a shim to intercept calls to the drawing function. ### Claim 17 (depends on 16) The system of claim 16, wherein the shim is configured to handle the calls to the drawing function or redirect the calls to the drawing function to another function. ### Claim 18 (depends on 13) The system of claim 13, wherein at least one drawing function comprises a drawing method. ### Claim 19 (depends on 13) The system of claim 13, wherein modifying the subset of drawing functions comprises associating image data with a resource name. ### Claim 20 A non-transitory computer readable medium storing instructions that upon execution by one or more computers cause the one or more computers to perform operations comprising: initiating, at a client device, a user session with one or more user interfaces of an application running on the client device; initiating, at the client device, an instrumentation module for collecting event data for use in recreating playback of the user session at a computing device different from the client device; identifying a set of drawing functions of the application; modifying at least a subset of the drawing functions to send drawing data related to objects being drawn to a display of the client device to the instrumentation module, wherein the drawing data for at least one drawing function includes data that enables another computing system to redraw the object using data indicating the drawing function; for each frame of multiple frames of the user interface displayed during the user session: obtaining, by the instrumentation module, data representing a plurality of views drawn to a display of the client device to generate display of the frame; and obtaining, by the instrumentation module and for each view of one or more of the plurality of views, data specifying one or more drawing operations processed by the application to draw content of the view on the display of the client device, wherein, for views that include a modified drawing function of the subset of drawing functions, the data includes the drawing data received from the modified drawing function; generating the event data comprising, for each one or more of the multiple frames, the data representing the plurality of views and the data specifying the one or more drawing operations; and sending the event data to the computing device.