# US 2019/0197063 A1 · Artificial intelligence for product data extraction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0197063 A1
- Document ID: US20190197063A1
- Status: Published application
- Kind: A1
- Application number: 16/288,059
- Filed: 2019-02-27
- Published: 2019-06-27
- Earliest claimed: 2019-02-19
- Assignees: Semantics3 Inc.
- Applicants: Semantics3 Inc.
- Inventors: Praveen Sekar; Govind Chandrasekhar; Ramanan Balakrishnan; Srinivasan Kidambi; Vinoth Gopinathan; Varun Sivamani
- Web page: https://patentgrep.com/patent/US20190197063A1

## CPC

- G06F 16/951: Indexing; Web crawling techniques
- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 20/00: Machine learning
- G06Q 30/0627: {by specifying product or service characteristics, e.g. product dimensions}
- G06N 5/02: Knowledge representation; Symbolic representation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A computer system and method may be used to generate a product catalog from one or more websites. One or more product pages on the websites may be identified and parsed. Attribute information may be identified in each page. Moreover, one or more automated interactions may be performed to generate page variations and identify attribute values. The attribute information and attribute values may be stored as structured data in a database.

## Claims

### Claim 1

A computer-implemented method for extracting content from a web page comprising:
crawling a website to identify a plurality of product pages, the product pages comprising base product pages comprising data about a product;
applying an unsupervised content extraction model to the product pages to generate a first set of rules for extracting product attributes;
filtering the first set of rules for extracting product attributes to create a second set of rules for extracting product attributes;
applying the second set of rules for extracting product attributes to the product pages;
identifying a plurality of interface elements on the product pages;
applying an automated process to systematically activate each of the plurality of interface elements to generate product page variations;
generating differences between the product page variations and the base product pages and analyzing the generated differences to identify a plurality of attribute values for each of the plurality of product attributes;
normalizing the plurality of product attributes and attribute values;
storing in a searchable database one or more product identifiers, product attributes, and attribute values.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
crawling the web site to visit a plurality of linked web pages and extract a plurality of URLs from the linked web pages;
clustering the extracted URLs based on similarity to generate a plurality of URL clusters;
generating a plurality of URL templates based on the URL clusters;
tagging a first set of URL templates as product page URLs;
prioritizing the product page URLs while crawling the website to identify the plurality of product pages.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
crawling the website to identify the plurality of product pages by using a reinforcement learning algorithm, where the reinforcement learning algorithm is rewarded for crawling a product page and penalized for visiting a non-product page.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the unsupervised content extraction model is a machine learning model.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the unsupervised content extraction model is trained to accept as input a plurality of HTML elements and select one of the HTML elements as a product attribute.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, further comprising:
extracting a plurality of HTML elements from a web page and extracting features from each of the HTML elements;
inputting the extracted features to the unsupervised content extraction model to select one of the plurality of HTML elements as a product attribute.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the first set of rules and second set of rules for extracting product attributes comprise regular expressions.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
providing a master list of product attributes and attribute values and normalizing the product attributes and attributes values to the master list.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the searchable database comprises a graph, the graph comprising nodes representing products and edges representing common attributes between connected nodes.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, further comprising:
training an embedding generation model to generate tensors for each product in the searchable database, where the distance between tensors represents the similarity of the products in at least one dimension.

### Claim 11

A non-transitory computer-readable medium comprising instructions for extracting content from a web page, the non-transitory computer-readable medium comprising instructions for:
crawling a website to identify a plurality of product pages, the product pages comprising base product pages comprising data about a product;
applying an unsupervised content extraction model to the product pages to generate a first set of rules for extracting product attributes;
filtering the first set of rules for extracting product attributes to create a second set of rules for extracting product attributes;
applying the second set of rules for extracting product attributes to the product pages;
identifying a plurality of interface elements on the product pages;
applying an automated process to systematically activate each of the plurality of interface elements to generate product page variations;
generating differences between the product page variations and the base product pages and analyzing the generated differences to identify a plurality of attribute values for each of the plurality of product attributes;
normalizing the plurality of product attributes and attribute values;
storing in a searchable database one or more product identifiers, product attributes, and attribute values.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, further comprising instructions for:
crawling the web site to visit a plurality of linked web pages and extract a plurality of URLs from the linked web pages;
clustering the extracted URLs based on similarity to generate a plurality of URL clusters;
generating a plurality of URL templates based on the URL clusters;
tagging a first set of URL templates as product page URLs;
prioritizing the product page URLs while crawling the website to identify the plurality of product pages.

### Claim 13 (depends on 11)

The non-transitory computer-readable medium of claim 11, further comprising instructions for:
crawling the website to identify the plurality of product pages by using a reinforcement learning algorithm, where the reinforcement learning algorithm is rewarded for crawling a product page and penalized for visiting a non-product page.

### Claim 14 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the unsupervised content extraction model is a machine learning model.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the unsupervised content extraction model is trained to accept as input a plurality of HTML elements and select one of the HTML elements as a product attribute.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, further comprising instructions for:
extracting a plurality of HTML elements from a web page and extracting features from each of the HTML elements;
inputting the extracted features to the unsupervised content extraction model to select one of the plurality of HTML elements as a product attribute.

### Claim 17 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the first set of rules and second set of rules for extracting product attributes comprise regular expressions.

### Claim 18 (depends on 11)

The non-transitory computer-readable medium of claim 11, further comprising instructions for:
providing a master list of product attributes and attribute values and normalizing the product attributes and attributes values to the master list.

### Claim 19 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the searchable database comprises a graph, the graph comprising nodes representing products and edges representing common attributes between connected nodes.

### Claim 20 (depends on 11)

The non-transitory computer-readable medium of claim 11, further comprising instructions for:
training an embedding generation model to generate tensors for each product in the searchable database, where the distance between tensors represents the similarity of the products in at least one dimension.
