# US 10,482,167 B2 · Crowd-source as a backup to asynchronous identification of a type of form and relevant fields in a credential-seeking web page > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,167 B2 - Document ID: US10482167B2 - Status: Granted - Kind: B2 - Application number: 14/864,448 - Filed: 2015-09-24 - Published: 2017-03-30 - Granted: 2019-11-19 - Earliest claimed: 2015-09-24 - Assignees: McAfee, Inc. - Applicants: McAfee, LLC - Inventors: Nicolas Lupien; Michael Lakhia; Hubert Gagnon-Lamonde - Web page: https://patentgrep.com/patent/US10482167B2 ## CPC - G06F 40/169: Annotation, e.g. comment data or footnotes - G06F 16/951: Indexing; Web crawling techniques - G06F 40/174: Form filling; Merging - G06F 40/226: Validation - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - H04L 67/5682: Policies or rules for updating, deleting or replacing the stored data - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A technique allows a credential manager application on a client computer system to identify fields and forms on a web page. An analysis server may automatically crawl web pages and identify the fields and form, then push the information to the client computer system for use by the credential manager. If the credential manager discovers the information is not available, the credential manager may analyze the web form to discover the fields and form information, then provide the discovered information to the analysis server for providing to other client computers. The analysis server may use crowd-sourcing for asynchronous verification of field and form information discovered by the analysis server or provided by the client computer. ## Claims ### Claim 1 A storage device comprising instructions that, when executed by a machine, cause the machine to at least: discover field and form information for a web page of a web site, the web page accessed over a network by an analysis server; create an annotated image of the web page to mark a position for a field of the field and form information, the annotated image created without displaying the web page to a user, the annotated image including the field and form information; transmit the annotated image with the field and form information to a crowd-sourcing service; accept the field and form information as validated field and form information responsive to a positive result from the crowd-sourcing service, the positive result indicating that the marked position of the field is correct; receive corrected field and form information from a human reviewer responsive to a negative result from the crowd-sourcing service, the negative result indicating that the marked position of the field is incorrect; send the annotated image with the field and form information to a human arbiter responsive to a mixed result from the crowd-sourcing service, and receive either corrected field and form information from the human arbiter responsive to a negative decision from the human arbiter or validated field and form information responsive to a positive decision from the human arbiter, the mixed result indicating that a minority of votes received by the crowd-sourcing service identify the marked position of the field as incorrect; send the at least one of validated or corrected field and form information from the analysis server to a credential manager application; instruct the credential manager application to cache the at least one of the validated or corrected field and form information, and the marked position of the field, in a cache memory; and instruct the credential manager application to insert credentials into a web page based on the at least one of the validated or corrected field and form information stored in the cache memory. ### Claim 2 (depends on 1) The storage device of claim 1, wherein the instructions, when executed, cause the machine to instruct the credential manager application to invalidate some or all of the cached field and form information maintained by the credential manager application. ### Claim 3 (depends on 1) The storage device of claim 1, wherein the instructions, when executed, cause the machine to: request at least three crowd-source responses from the crowd-sourcing service; and consider the at least three crowd-source responses as a voting result. ### Claim 4 (depends on 1) The storage device of claim 1, wherein the instructions, when executed, cause the machine to: use computer vision to view the annotated image of the web page annotated corresponding to the field and form information; and return a decision based on the computer vision that indicates a confidence level that the field and form information is correct. ### Claim 5 (depends on 4) The storage device of claim 4, wherein the instructions, when executed, cause the machine to pass the annotated image to the human reviewer responsive to the confidence level being lower than a threshold. ### Claim 6 (depends on 1) The storage device of claim 1, wherein the instructions, when executed, cause the machine to employ a web crawler to examine web pages. ### Claim 7 (depends on 6) The storage device of claim 6, wherein the web crawler is to crawl a set of web pages. ### Claim 8 A computer system for determining web form information in a web page for a web site, the computer system comprising: one or more processors; and memory to store instructions that, when executed by the one or more processors, cause at least one of the one or more of the processors to at least: discover field and form information for a web page of a web site, the web page accessed over a network by an analysis server; create an annotated image of the web page to mark a position for a field of the field and form information, the annotated image created without displaying the web page to a user, the annotated image including the field and form information; transmit the annotated image including the field and form information to a crowd-sourcing service; accept the field and form information as validated field and form information responsive to a positive result from the crowd-sourcing service, the positive result indicating that the marked position of the field is correct; receive corrected field and form information from a human reviewer responsive to a negative result from the crowd-sourcing service, the negative result indicating that the marked position of the field is incorrect; send the annotated image with field and form information to a human arbiter responsive to a mixed result from the crowd-sourcing service, and receive either corrected field and form information from the human arbiter responsive to a negative decision from the human arbiter or validated field and form information responsive to a positive decision from the human arbiter, the mixed result indicating that a minority of votes received by the crowd-sourcing service identify the marked position of the field as incorrect; send the at least one of the validated or corrected field and form information from the analysis server to a credential manager application; instruct the credential manager application to cache the at least one of the validated or corrected field and form information, and the marked position of the field, in a cache memory; and instruct the credential manager application to insert credentials into a web page based on the at least one of the validated or corrected field and form information stored in the cache memory. ### Claim 9 (depends on 8) The computer system of claim 8, wherein the at least one of the one or more processors is to instruct the credential manager application to invalidate at least a portion of the cached field and form information maintained by the credential manager application. ### Claim 10 (depends on 8) The computer system of claim 8, wherein at least one of the one or more processors is to at least: request at least three crowd-source responses from the crowd-sourcing service; and consider the at least three crowd-source responses as a voting result. ### Claim 11 (depends on 8) The computer system of claim 8, wherein at least one of the one or more processors is to at least: use computer vision to view the annotated image of the web page annotated corresponding to the field and form information; and return a decision based on the computer vision that indicates a confidence level that the field and form information is correct. ### Claim 12 (depends on 11) The computer system of claim 11, wherein at least one of the one or more processors is to pass the annotated image to the human reviewer responsive to the confidence level being lower than a threshold. ### Claim 13 (depends on 8) The computer system of claim 8, wherein at least one of the one or more processors is to receive the web page by employing a web crawler to examine web pages. ### Claim 14 (depends on 13) The computer system of claim 13, wherein the web crawler is to crawl a set of web pages. ### Claim 15 A method for determining web form information in a web page for a web site, the method comprising: discovering field and form information for a web page of a web site, the web page accessed over a network by an analysis server; creating an annotated image of the web page to mark a position for a field of the field and form information, the annotated image created without displaying the web page to a user, the annotated image including the field and form information; transmitting the annotated image with the field and form information to a crowd-sourcing service; accepting the field and form information as validated field and form information responsive to a positive result from the crowd-sourcing service, the positive result indicating that the marked position of the field is correct; receiving corrected field and form information from a human reviewer responsive to a negative result from the crowd-sourcing service, the negative result indicating that the marked position of the field is incorrect; sending the annotated image with the field and form information to a human arbiter responsive to a mixed result from the crowd-sourcing service, and receive either corrected field and form information from the human arbiter responsive to a negative decision from the human arbiter or validated field and form information responsive to a positive decision from the human arbiter, the mixed result indicating that a minority of votes received by the crowd-sourcing service identify the marked position of the field as incorrect; sending the at least one of the validated or corrected field and form information from the analysis server to a credential manager application; instructing the credential manager application to cache the at least one of the validated or corrected field and form information, and the position of the field, in a cache memory; and instructing the credential manager application to insert credentials into a web page based on the at least one of the validated or corrected field and form information stored in the cache memory. ### Claim 16 (depends on 15) The method of claim 15, further including instructing the credential manager application to invalidate at least a portion of the cached field and form information maintained by the credential manager application. ### Claim 17 (depends on 15) The method of claim 15, wherein the validating of the field and form information using crowd-sourcing service includes: requesting at least three crowd-source responses from the crowd-sourcing service; and considering the at least three crowd-source responses as a voting result. ### Claim 18 (depends on 15) The method of claim 15, further including: using computer vision to view the annotated image of the web page annotated corresponding to the field and form information; and return a decision based on the computer vision that indicates a confidence level that the field and form information is correct. ### Claim 19 (depends on 15) The method of claim 15, wherein the receiving of the web page includes employing a web crawler to examine web pages.