# US 12,118,351 B2 · Automatic capture of user interface screenshots for software product documentation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,351 B2 - Document ID: US12118351B2 - Status: Granted - Kind: B2 - Application number: 17/394,468 - Filed: 2021-08-05 - Published: 2023-02-09 - Granted: 2024-10-15 - Earliest claimed: 2021-08-05 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Shi Kun Li; Jun Qian Zhou; Dan Zhang; Xiao Feng Ji - Web page: https://patentgrep.com/patent/US12118351B2 ## CPC - G06F 8/73: Program documentation - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2323: based on graph theory, e.g. minimum spanning trees [MST] or graph cuts - G06F 18/2431: Multiple classes - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Embodiments of the invention are directed to automatically capturing user interface screenshots for use in documentation of a software product. Aspects include identifying a user interface window of the software product and creating a degree-of-completion graph for the user interface window. Aspects also include capturing a plurality of screenshots of the user interface window during use of the software product and calculating a degree-of-completion percentage for each of the plurality of screenshots. Aspects further include identifying a subset of the plurality of screenshots to be included in the software product documentation based on the degree-of-completion percentage. ## Claims ### Claim 1 A method for automatically capturing user interface screenshots for use in documentation of a software product, the method comprising: identifying a user interface window of the software product; creating a first degree-of-completion graph for the user interface window, wherein the first degree-of-completion graph includes two nodes representing each user interface control of the user interface window; capturing a plurality of screenshots of the user interface window during use of the software product; calculating a degree-of-completion percentage for each of the plurality of screenshots, wherein the degree-of-completion percentage for each of the plurality of screenshots is calculated by inputting the first degree-of-completion graph into a three-layer graph convolutional network, wherein the degree-of-completion percentage calculated for each of the plurality of screenshots is based on embeddings representing a current degree-of- completion graph, a second degree-of-completion graph having each input-free node connected to a window node, and a third degree-of-completion graph having each input node belonging to all user interface controls connected to the window node; and identifying a subset of the plurality of screenshots to be included in the documentation of the software product based on the degree-of-completion percentage calculated for each of the plurality of screenshots. ### Claim 2 (depends on 1) The method of claim 1, wherein the first degree-of-completion graph is an undirected and weighted graph that is created by identifying and grouping user interface elements of the user interface window. ### Claim 3 (depends on 1) The method of claim 1, wherein the capturing of the plurality of screenshots of the user interface window during the use of the software product includes capturing screenshots on a periodic basis while the user interface window is open. ### Claim 4 (depends on 1) The method of claim 1, wherein the capturing of the plurality of screenshots of the user interface window during the use of the software product includes capturing screenshots each time a user interface event occurs for the user interface window. ### Claim 5 (depends on 1) The method of claim 1, further comprising: obtaining documentation of a prior version of the software product and identifying prior screenshots included in the documentation of the prior version; classifying each of the identified prior screenshots as belonging to a user interface window category; determining a particular user interface window category for the user interface window; and calculating a degree-of-completion percentage for each of the identified prior screenshots that belong to the particular user interface window category. ### Claim 6 (depends on 5) The method of claim 5, wherein the subset of the plurality of screenshots to be included in the documentation includes a plurality of screenshots that have a highest cosine similarity. ### Claim 7 A system comprising: a processor communicatively coupled to a memory, the processor configured to: identify a user interface window of a software product; create a first degree-of-completion graph for the user interface window, wherein the first degree-of-completion graph includes two nodes representing each user interface control of the user interface window; capture a plurality of screenshots of the user interface window during use of a software product; calculate a degree-of-completion percentage for each of the plurality of screenshots, wherein the degree-of-completion percentage for each of the plurality of screenshots is calculated by inputting the first degree-of-completion graph into a three-layer graph convolutional network, wherein the degree-of-completion percentage calculated for each of the plurality of screenshots is based on embeddings representing a current degree-of-completion graph, a second degree-of-completion graph having each input-free node connected to a window node, and a third degree-of-completion graph having each input node belonging to all user interface controls connected to the window node; and identify a subset of the plurality of screenshots to be included in documentation of the software product based on the degree-of-completion percentage calculated for each of the plurality of screenshots. ### Claim 8 (depends on 7) The system of claim 7, wherein the first degree-of-completion graph is an undirected and weighted graph that is created by identifying and grouping user interface elements of the user interface window. ### Claim 9 (depends on 7) The system of claim 7, wherein the capturing of the plurality of screenshots of the user interface window during the use of the software product includes capturing screenshots on a periodic basis while the user interface window is open. ### Claim 10 (depends on 7) The system of claim 7, wherein the capturing of the plurality of screenshots of the user interface window during the use of the software product includes capturing screenshots each time a user interface event occurs for the user interface window. ### Claim 11 (depends on 7) The system of claim 7, wherein the processor is further configured to: obtain documentation of a prior version of the software product and identifying prior screenshots included in the documentation of the prior version; classify each of the identified prior screenshots as belonging to a user interface window category; determine a particular user interface window category for the user interface window; and calculate a degree-of-completion percentage for each of the identified screenshots that belong to the particular user interface window category. ### Claim 12 A computer program product for automatically capturing user interface screenshots for use in software product documentation comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising: identifying a user interface window of a software product; creating a first degree-of-completion graph for the user interface window, wherein the first degree-of-completion graph includes two nodes representing each user interface control of the user interface window; capturing a plurality of screenshots of the user interface window during use of the software product; calculating a degree-of-completion percentage for each of the plurality of screenshots, wherein the degree-of-completion percentage for each of the plurality of screenshots is calculated by inputting the first degree-of-completion graph into a three-layer graph convolutional network, wherein the degree-of-completion percentage calculated for each of the plurality of screenshots is based on embeddings representing a current degree-of-completion graph, a second degree-of-completion graph having each input-free node connected to a window node, and a third degree-of-completion graph having each input node belonging to all user interface controls connected to the window node; and identifying a subset of the plurality of screenshots to be included in the software product documentation based on the degree-of-completion percentage calculated for each of the plurality of screenshots. ### Claim 13 (depends on 12) The computer program product of claim 12, wherein the first degree-of-completion graph is an undirected and weighted graph that is created by identifying and grouping user interface elements of the user interface window. ### Claim 14 (depends on 12) The computer program product of claim 12, wherein the calculating of the degree-of-completion percentage for each of the plurality of screenshots includes analyzing the first degree-of-completion graph calculated for each of the plurality of screenshots. ### Claim 15 (depends on 12) The computer program product of claim 12, wherein the capturing of the plurality of screenshots of the user interface window during the use of the software product includes capturing screenshots on a periodic basis while the user interface window is open. ### Claim 16 (depends on 12) The computer program product of claim 12, wherein the method further comprises: obtaining documentation of a prior version of the software product and identifying prior screenshots included in the documentation of the prior version; classifying each of the identified prior screenshots as belonging to a user interface window category; determining a particular user interface window category for the user interface window; and calculating a degree-of-completion percentage for each of the identified prior screenshots that belong to the particular user interface window category.