# US 9,436,577 B2 · System and method for generating a code execution timeline from an executing program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,436,577 B2 - Document ID: US9436577B2 - Status: Granted - Kind: B2 - Application number: 14/087,586 - Filed: 2013-11-22 - Published: 2015-05-28 - Granted: 2016-09-06 - Earliest claimed: 2013-11-22 - Assignees: Nintendo Co., Ltd. - Applicants: Nintendo Co., Ltd. - Inventors: Steve Rabin; Fernando Silva - Web page: https://patentgrep.com/patent/US9436577B2 ## CPC - A63F 13/60: Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor - G06F 11/3466: {Performance evaluation by tracing or monitoring} - G06F 11/30: Monitoring - G06F 11/3636: {by tracing the execution of the program} - A63F 2300/209: characterized by low level software layer, relating to hardware management, e.g. Operating System, Application Programming Interface - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The present technology is directed to a non-limiting system and method for generating an abstract code execution timeline from repetitive programs, such as video games and simulations. As a non-limiting example, a statistical sampling profile is first taken of the running program, and then the data is converted into several representations. By selecting representative functions based on sampling frequency and call tree hierarchy and then sorting those functions chronologically and hierarchically, an abstract code execution timeline can be generated that summarizes execution within a single repetitive time period, such as a video frame. This timeline is called “Story Mode” since it tells a story of how the program executes over the course of a frame. This abstract code execution timeline can then be used to facilitate program comprehension, primarily for the purposes of modifying the code to improve execution speed. ## Claims ### Claim 1 A method performed by at least one processor for analyzing video game execution, the method comprising: maintaining a record of execution of one or more functions of the video game during a frame of video game execution; selecting a subset of the one or more functions based on at least activity of the record of execution of the selected functions; sorting the selected subset of functions based on both chronological order of execution and hierarchical function calling relationships indicated by the record of function execution; and generating, using the at least one processor, a timeline display having a first axis for the selected and sorted subset of functions and having a second axis for graphically depicting function execution over time, the timeline display showing function clusters of video game execution over time in an order showing diagonally along the timeline display relative to the first and second axes. ### Claim 2 (depends on 1) The method according to claim 1, wherein the sample runs for each function are displayed in a first color when the function is being accessed for a portion of time during the frame of program execution and a second color when the function is not being accessed for the period of time during the frame of program execution. ### Claim 3 A method implemented in an information processing apparatus having one or more processors for generating a code execution timeline, the method comprising: recording, from an interactive program executing one or more functions, execution histories of the one or more functions being accessed in the interactive program, the interactive program configured for execution on, at least, a video game system; sorting the functions based at least in part on a criteria related to function execution; and generating, via the one or more processors, a timeline display having a first axis for the sorted functions and having a second axis for graphically depicting function execution over time, the timeline display showing, at least, function clusters of program execution, associated with the interactive program, over time in an order showing diagonally along the timeline display relative to the first and second axes. ### Claim 4 (depends on 3) The method of claim 3, further comprising: displaying a list of functions on a first axis of the timeline display; and displaying, next to each respective function along a second axis of the timeline display, at least one execution event for each function in the list of functions, the generated timelines displayed such that the timelines contain the list of functions and the at least one execution event for each function. ### Claim 5 (depends on 4) The method of claim 4, further comprising: selecting a group of functions from the one or more functions; displaying the selected group of functions in the timeline display; and modifying the timeline display to remove display of one or more functions having similar execution from the selected group of functions. ### Claim 6 (depends on 5) The method of claim 5, wherein the one or more functions having similar execution call one or more similar child functions. ### Claim 7 (depends on 4) The method of claim 4, further comprising: selecting a representative group of functions from the one or more functions; and modifying the timeline display to exclude parent functions from the selected representative group of functions. ### Claim 8 (depends on 4) The method of claim 4, further comprising: selecting a representative group of functions from the one or more functions; and modifying the timeline display to include parent functions from the selected representative group of functions. ### Claim 9 (depends on 4) The method of claim 4, wherein each function in the timeline display can be displayed in a color different from at least one other function. ### Claim 10 (depends on 3) The method of claim 3, wherein the functions are sorted based on both a chronological order of execution and hierarchical function calling relationship indicated by the record of function execution. ### Claim 11 A non-transitory computer-readable storage medium comprising instructions which, when executed by one or more processors of an information processing apparatus, cause the information processing apparatus to: record, from an interactive program executing one or more functions, execution histories of the one or more functions being accessed in the interactive program, the interactive program configured to execute on a system configured for video game functionality; sort and filter the functions based at least in part on a criteria related to function execution; and generate a timeline display having a first axis for the sorted and filtered functions and having a second axis for graphically depicting function execution over time, the timeline display showing, at least, function clusters of program execution, associated with the interactive program, over time in an order showing diagonally along the timeline display relative to the first and second axes. ### Claim 12 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the information processing apparatus is further caused to: display a list of functions on a first axis of the timeline display; and display, next to each respective function along a second axis of the timeline display, at least one execution event for each function in the list of functions, the generated timelines displayed such that the timelines contain the list of functions and the at least one execution event for each function. ### Claim 13 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the information processing apparatus is further caused to: select a group of functions from the one or more functions; display the selected group of functions in the timeline display; and modify the timeline display to remove display of one or more functions having similar execution from the selected group of functions. ### Claim 14 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein the one or more functions having similar execution call one or more similar child functions. ### Claim 15 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the information processing apparatus is further caused to: select a representative group of functions from the one or more functions; and modify the timeline display to exclude parent functions from the selected representative group of functions. ### Claim 16 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the information processing apparatus is further caused to: select a representative group of functions from the one or more functions; and modify the timeline display to include parent functions from the selected representative group of functions. ### Claim 17 A system, comprising: a processing system having a memory and one or more processors, the processing system configured to: record, from an interactive program executing one or more functions, execution histories of the one or more functions being accessed in the interactive program, the interactive program configured for execution using an information processing system that is capable of providing video game functionality, selectively sort and filter the functions based at least in part on a criteria related to function execution, and generate a timeline display having a first axis for the sorted and filtered functions and having a second axis for graphically depicting function execution over time, the timeline display showing, at least, function clusters of program execution, associated with the interactive program, over time in an order showing diagonally along the timeline display relative to the first and second axes. ### Claim 18 (depends on 17) The system of claim 17, wherein the processing system is further configured to: display a list of functions on a first axis of the timeline display; and display, next to each respective function along a second axis of the timeline display, at least one execution event for each function in the list of functions, the generated timelines displayed such that the timelines contain the list of functions and the at least one execution event for each function. ### Claim 19 (depends on 18) The system of claim 18, wherein the processing system is further configured to: select a group of functions from the one or more functions; display the selected group of functions in the timeline display; and modify the timeline display to remove display of one or more functions having similar execution from the selected group of functions. ### Claim 20 (depends on 19) The system of claim 19, wherein the one or more functions having similar execution call one or more similar child functions. ### Claim 21 (depends on 18) The system of claim 18, wherein the processing system is further configured to: select a representative group of functions from the one or more functions; and modify the timeline display to exclude parent functions from the selected representative group of functions. ### Claim 22 (depends on 18) The system of claim 18, wherein the processing system is further configured to: select a representative group of functions from the one or more functions; and modify the timeline display to include parent functions from the selected representative group of functions.