# US 10,319,041 B2 · Automated financial data aggregation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,319,041 B2 - Document ID: US10319041B2 - Status: Granted - Kind: B2 - Application number: 14/790,509 - Filed: 2015-07-02 - Published: 2015-10-29 - Granted: 2019-06-11 - Earliest claimed: 2013-03-11 - Assignees: Yodlee, Inc. - Applicants: Yodlee, Inc. - Inventors: Rohit Chourasia; Shubha Pant - Web page: https://patentgrep.com/patent/US10319041B2 ## CPC - G06Q 40/12: Accounting - G06Q 30/0201: Market modelling; Market analysis; Collecting market data - G06Q 40/00: Finance; Insurance; Tax strategies; Processing of corporate or income taxes - G06F 16/00: Information retrieval; Database structures therefor; File system structures therefor - G06F 16/958: Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data aggregation. The methods, systems, and apparatus include determining whether a site-specific script for extracting financial data from a particular financial institution website is available; in response to determining that a site-specific script for extracting financial data from the particular financial institution website is not available, generating a site map of web pages and web page segments in the financial institution website, wherein the site map is generated based on at least in part on a statistical analysis of web pages and web page segments that are not in the financial institution website; generating, based on the site map of the financial institution website, a site-specific script for extracting financial data from the financial institution website; and extracting, for one or more users, financial data from the particular financial institution website using the generated site-specific script. ## Claims ### Claim 1 A computer-implemented method, comprising: analyzing, in a computer, data for each webpage of a plurality of webpages that are not in a financial institution website; generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage of the one or more identified web pages and (b) the analyzed data; generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script. ### Claim 2 (depends on 1) The method of claim 1, wherein determining, respective categorizations for the one or more identified web pages in the financial institution website further comprises: determining a respective plurality of scores for each web page in the one or more identified web pages, each score in the plurality of scores indicating a confidence that the web page corresponds to a particular category; determining, for each web page in the one or more identified web pages, whether a score in the respective plurality of scores satisfies a threshold; and in response to determining that a score in the respective plurality of scores for a web page satisfies a threshold, associating the web page in the one or more web pages with a particular category corresponding to the score. ### Claim 3 (depends on 2) The method of claim 2, further comprising: in response to determining that no score in the respective plurality of scores for a web page satisfies a threshold, providing the web page to a user for manual categorization; and associating the web page with a category specified by the user. ### Claim 4 (depends on 1) The method of claim 1, wherein analyzing data for each webpage of the plurality of webpages that are not in a financial institution website comprises: generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web pages to respective categories of the web pages. ### Claim 5 (depends on 4) The method of claim 4, wherein determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage in the one or more identified web pages and (b) the analyzed data comprises: using the data model to categorize web pages in the financial institution website. ### Claim 6 (depends on 1) The method of claim 1, wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input. ### Claim 7 (depends on 1) The method of claim 1, further comprising: determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available. ### Claim 8 A computer-implemented method, comprising: analyzing, in a computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website; generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web page portions in one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data; generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script. ### Claim 9 (depends on 8) The method of claim 8, wherein determining, respective categorizations for the one or more identified web pages portions in the one or more web pages in the financial institution website further comprises: determining a respective plurality of scores for each web page portion in the one or more web pages, each score in the plurality of scores indicating a confidence that a web page portion corresponds to a particular category; determining, for each web page portion, whether a score in the respective plurality of scores satisfies a threshold; and in response to determining that a score in the respective plurality of scores for a web page portion satisfies a threshold, associating the web page portion with a particular category corresponding to the score. ### Claim 10 (depends on 9) The method of claim 9, further comprising: in response to determining that no score in the respective plurality of scores for a web page portion satisfies a threshold, providing the web page portion to a user for manual categorization; and associating the web page with a category specified by the user. ### Claim 11 (depends on 8) The method of claim 8, wherein analyzing, in the computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website comprises: generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web page portions to respective categories of the web page portions. ### Claim 12 (depends on 11) The method of claim 11, wherein determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data comprises: using the data model to categorize web page portions in the financial institution website. ### Claim 13 (depends on 8) The method of claim 8, wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input. ### Claim 14 (depends on 8) The method of claim 8, further comprising: determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available. ### Claim 15 A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising: analyzing, in a computer, data for each webpage of a plurality of webpages that are not in a financial institution website; generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage of the one or more identified web pages and (b) the analyzed data; generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script. ### Claim 16 (depends on 15) The computer storage medium of claim 15, wherein determining, respective categorizations for the one or more identified web pages in the financial institution web site further comprises: determining a respective plurality of scores for each web page in the one or more identified web pages, each score in the plurality of scores indicating a confidence that the web page corresponds to a particular category; determining, for each web page in the one or more identified web pages, whether a score in the respective plurality of scores satisfies a threshold; and in response to determining that a score in the respective plurality of scores for a web page satisfies a threshold, associating the web page in the one or more web pages with a particular category corresponding to the score. ### Claim 17 (depends on 16) The computer storage medium of claim 16, wherein the operations further comprise: in response to determining that no score in the respective plurality of scores for a web page satisfies a threshold, providing the web page to a user for manual categorization; and associating the web page with a category specified by the user. ### Claim 18 (depends on 15) The computer storage medium of claim 15, wherein analyzing, in the computer, data for each webpage of the plurality of webpages that are not in a financial institution website comprises: generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web pages to respective categories of the web pages. ### Claim 19 (depends on 18) The computer storage medium of claim 18, wherein determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage in the one or more identified web pages and (b) the analyzed data comprises: using the data model to categorize web pages in the financial institution website. ### Claim 20 (depends on 15) The computer storage medium of claim 15, wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input. ### Claim 21 (depends on 15) The computer storage medium of claim 15, wherein the operations further comprise: determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available. ### Claim 22 A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising: analyzing, in a computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website; generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web page portions in one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data; generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script. ### Claim 23 (depends on 22) The computer storage medium of claim 22, wherein determining, respective categorizations for the one or more identified web pages portions in the one or more web pages in the financial institution website further comprises: determining a respective plurality of scores for each web page portion in the one or more web pages, each score in the plurality of scores indicating a confidence that a web page portion corresponds to a particular category; determining, for each web page portion, whether a score in the respective plurality of scores satisfies a threshold; and in response to determining that a score in the respective plurality of scores for a web page portion satisfies a threshold, associating the web page portion with a particular category corresponding to the score. ### Claim 24 (depends on 23) The computer storage medium of claim 23, wherein the operations further comprise: in response to determining that no score in the respective plurality of scores for a web page portion satisfies a threshold, providing the web page portion to a user for manual categorization; and associating the web page with a category specified by the user. ### Claim 25 (depends on 22) The computer storage medium of claim 22, wherein analyzing, in the computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution web site comprises: generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web page portions to respective categories of the web page portions. ### Claim 26 (depends on 25) The computer storage medium of claim 25, wherein determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data comprises: using the data model to categorize web page portions in the financial institution website. ### Claim 27 (depends on 22) The computer storage medium of claim 22, wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input. ### Claim 28 (depends on 22) The computer storage medium of claim 22 wherein the operations further comprise: determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available. ### Claim 29 A system comprising one or more computers programmed to perform operations comprising: analyzing, in a computer, data for each webpage of a plurality of webpages that are not in a financial institution website; generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage of the one or more identified web pages and (b) the analyzed data; generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script. ### Claim 30 (depends on 29) The system of claim 29, wherein determining, respective categorizations for the one or more identified web pages in the financial institution website further comprises: determining a respective plurality of scores for each web page in the one or more identified web pages, each score in the plurality of scores indicating a confidence that the web page corresponds to a particular category; determining, for each web page in the one or more identified web pages, whether a score in the respective plurality of scores satisfies a threshold; and in response to determining that a score in the respective plurality of scores for a web page satisfies a threshold, associating the web page in the one or more web pages with a particular category corresponding to the score. ### Claim 31 (depends on 30) The system of claim 30, wherein the operations further comprise: in response to determining that no score in the respective plurality of scores for a web page satisfies a threshold, providing the web page to a user for manual categorization; and associating the web page with a category specified by the user. ### Claim 32 (depends on 29) The system of claim 29, wherein analyzing, in the computer, data for each webpage of the plurality of webpages that are not in a financial institution website comprises: generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web pages to respective categories of the web pages. ### Claim 33 (depends on 32) The system of claim 32, wherein determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage in the one or more identified web pages and (b) the analyzed data comprises: using the data model to categorize web pages in the financial institution website. ### Claim 34 (depends on 29) The system of claim 29, wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input. ### Claim 35 (depends on 29) The system of claim 29, wherein the operations further comprise: determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available. ### Claim 36 A system comprising one or more computers programmed to perform operations comprising: analyzing, in a computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website; generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web page portions in one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data; generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script. ### Claim 37 (depends on 36) The system of claim 36, wherein determining, respective categorizations for the one or more identified web pages portions in the one or more web pages in the financial institution website further comprises: determining a respective plurality of scores for each web page portion in the one or more web pages, each score in the plurality of scores indicating a confidence that a web page portion corresponds to a particular category; determining, for each web page portion, whether a score in the respective plurality of scores satisfies a threshold; and in response to determining that a score in the respective plurality of scores for a web page portion satisfies a threshold, associating the web page portion with a particular category corresponding to the score. ### Claim 38 (depends on 37) The system of claim 37, wherein the operations further comprise: in response to determining that no score in the respective plurality of scores for a web page portion satisfies a threshold, providing the web page portion to a user for manual categorization; and associating the web page with a category specified by the user. ### Claim 39 (depends on 36) The system of claim 36, wherein analyzing, in the computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website comprises: generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web page portions to respective categories of the web page portions. ### Claim 40 (depends on 39) The system of claim 39, wherein determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data comprises: using the data model to categorize web page portions in the financial institution website. ### Claim 41 (depends on 36) The system of claim 36, wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input. ### Claim 42 (depends on 36) The system of claim 36 wherein the operations further comprise: determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available.