# US 2021/0240910 A1 · Automatically locating elements in user interfaces > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240910 A1 - Document ID: US20210240910A1 - Status: Published application - Kind: A1 - Application number: 16/779,390 - Filed: 2020-01-31 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: salesforce.com, inc. - Applicants: salesforce.com, inc. - Inventors: Jessica Lundin; David James Woodward; Alan Martin Ross; Brian J. Lonsdorf; Owen Winne Schoppe; Sönke Rohde - Web page: https://patentgrep.com/patent/US20210240910A1 ## CPC - G06F 8/38: for implementing user interfaces - G06F 40/106: Display of layout of documents; Previewing - G06F 8/34: Graphical or visual programming - G06F 9/451: Execution arrangements for user interfaces - G06F 40/151: Transformation - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Techniques are disclosed relating to determining locations of child elements in a user interface (UI), automatically updating the locations of the child elements in the UI, and generating an updated UI with the updated locations. A computer system may receive information specifying content of the UI and render an image of the UI based on the received information. Computer vision algorithms may be applied to the rendered of the UI to determine locations of elements in the UI. The locations of the elements may be updated to position the elements in selected locations relative to each other. The updated locations of the elements as determined by the computer vision algorithms may be implemented to generate updating information specifying content for the updated UI. ## Claims ### Claim 1 A method, comprising: receiving, by a computer system, information specifying content of a user interface (UI); generating, by the computer system based on the received information, an image that is a rendering of the UI, wherein the image includes a rendering of at least two child elements in the UI; applying, by the computer system, a machine learning algorithm on the image to determine location information for content included within the at least two child elements, wherein the location information includes a position of the content inside the at least two child elements; updating, by the computer system, the location information for the content included within at least one of the at least two child elements until the content included within the at least two child elements is positioned in selected locations relative to each other in the image; and generating, by the computer system, updated information specifying the content of the UI based on the updated location information for the content included within the at least one child element. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the location information for the content included within the at least two child elements includes determining bounding regions for the content included within the at least two child elements that is visible in the image and determining location information for the bounding regions. ### Claim 3 (depends on 2) The method of claim 2, wherein the determined location information includes spacing information for the bounding regions in the image. ### Claim 4 (depends on 1) The method of claim 1, further comprising determining, by the application of the machine learning algorithm on the image, identifying information of the content included within the at least two child elements in the image, wherein determining the identification information includes determining a type of content for the content included within the at least two child elements. ### Claim 5 (canceled) ### Claim 6 (depends on 1) The method of claim 1, wherein positioning the content included within the at least two child elements in the selected locations relative to each other includes positioning the content included within the at least two child elements in a selected alignment relative to each other. ### Claim 7 (depends on 1) The method of claim 1, wherein the machine learning algorithm is a computer vision algorithm. ### Claim 8 (depends on 1) The method of claim 1, further comprising generating, by the computer system, a report of the updated location information for the content included within the at least one child element. ### Claim 9 (depends on 1) The method of claim 1, wherein updating the location information for the content included within the at least one of the at least two child elements until the content included within the at least two child elements are positioned in selected locations relative to each other in the image includes: (a) updating the location information for the content included within the at least one of the at least two child elements in the image; (b) determining, by applying the machine learning algorithm to the image, the updated location information for the content included within the at least two child elements in the image; determining, by the computer system, whether the content included within the at least two child elements are positioned in selected locations relative to each other in the image based on the updated location information; in response to the content included within the at least two child elements being determined to be positioned in the selected locations relative to each other in the image, generating the updated information specifying the content of the UI based on the updated location information for the content included within the at least one child element; and in response to the content included within the at least two child elements being determined to not be positioned in the selected locations relative to each other in the image, repeating (a) and (b) until the content included within the at least two child elements is determined to be positioned in the selected locations relative to each other in the image. ### Claim 10 A non-transitory computer-readable medium having program instructions stored thereon that are executable by a user interface (UI) system to perform operations comprising: receiving, by a computer system, information specifying content of a user interface (UI); generating, by the computer system based on the received information, an image that is a rendering of the UI, wherein the image includes a rendering of at least two child elements in the UI; applying, by the computer system, a machine learning algorithm on the image to determine location information for content included within the at least two child elements, wherein the location information includes a position of the content inside the at least two child elements; determining, by the computer system, a relative alignment for the content included within the at least two child elements based on the location information for the content included within the at least two child elements in the image; and generating, by the computer system, a report of the relative alignment determined for the content included within the at least two child elements. ### Claim 11 (depends on 10) The non-transitory computer-readable medium of claim 10, further comprising transmitting, by the computer system, the report to a computer system operating the UI. ### Claim 12 (depends on 10) The non-transitory computer-readable medium of claim 10, further comprising generating, by the computer system, updated information specifying the content of the UI based on the report. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, further comprising transmitting, by the computer system, the updated information specifying the content of the UI to a computer system operating the UI. ### Claim 14 (canceled) ### Claim 15 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein determining the relative alignment for the content included within the at least two child elements includes determining whether the content included within the at least two child elements are in a selected alignment, and wherein the report includes the determination on whether the at least two child elements are in the selected alignment. ### Claim 16 A method, comprising: receiving, by a computer system, an image that is a rendering of a user interface (UI), wherein the image includes a rendering of a plurality of child elements positioned within at least one outer bounding region in the UI, and wherein the image includes a rendering of content within the plurality of child elements; updating, by the computer system, the image by placing the content included within one or more of the child elements in selected locations in the image; and applying, by the computer system, a machine learning algorithm on the updated image to determine location information for the content included within the plurality of child elements in the updated image, wherein the location information includes a position of the content inside the plurality of child elements; and generating, by the computer system, information specifying content of the UI based on the location information for the content included within the plurality of child elements in the updated image. ### Claim 17 (depends on 16) The method of claim 16, wherein the information specifying the content of the UI generated by the computer system is generated independently of information specifying the content of the UI for the received image of the UI. ### Claim 18 (depends on 16) The method of claim 16, further comprising generating at least one additional UI using the information specifying the content of the UI. ### Claim 19 (depends on 16) The method of claim 16, wherein placing the content included within the one or more of the child elements in the selected locations in the image includes placing the content included within at least two child elements in selected locations relative to each other. ### Claim 20 (canceled) ### Claim 21 (depends on 2) The method of claim 2, wherein the bounding regions for the content in the at least two child elements that is visible in the image are determined independently of bounding regions for the at least two child elements defined by the information specifying content of the UI. ### Claim 22 (depends on 4) The method of claim 4, wherein determining the type of content includes determining whether the content is text content, image content, or shape content. ### Claim 23 (depends on 4) The method of claim 4, wherein at least one of the at least two child elements is an image element, and wherein determining the type of content included within the image element includes identifying image content and determining a confidence level for the identification of the image content.