System and method to identify machine-readable codes

A method comprising:

• downloading a plurality of pages from the World Wide Web using a web crawler, the plurality of pages downloaded based on a list of Uniform Resource Locators (URLs), the list comprising a seed and an additional URL identified from a hyperlink in the seed, the seed being a website selected according to one or more policies, at least one of the one or more policies based on a number of unknown machine-readable codes identified from pages previously downloaded from the website;
• identifying a machine-readable code included in a downloaded page of the plurality of downloaded pages;
• determining if a product description is available from the downloaded page for the identified machine-readable code;
• based on the product description being available, extracting the product description from the downloaded page;
• and
• storing a record of the machine-readable code and the product description.

A non-transitory machine-readable medium having instructions embodied thereon, the instructions executable by one or more machines to perform operations comprising:

• downloading a plurality of pages from the World Wide Web using a web crawler, the plurality of pages downloaded based on a list of Uniform Resource Locators (URLs), the list comprising a seed and an additional URL identified from a hyperlink in the seed, the seed being a website selected according to one or more policies, at least one of the one or more policies based on a number of unknown machine-readable codes identified from pages previously downloaded from the website;
• identifying a machine-readable code included in a downloaded page of the plurality of downloaded pages;
• determining if a product description is available from the downloaded page for the identified machine-readable code;
• based on the product description being available, extracting the product description from the downloaded page;
• and
• storing a record of the machine-readable code and the product description.