# US 9,519,718 B2 · Webpage information detection method and system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,519,718 B2
- Document ID: US9519718B2
- Status: Granted
- Kind: B2
- Application number: 13/997,251
- Filed: 2011-12-22
- Published: 2014-03-06
- Granted: 2016-12-13
- Earliest claimed: 2010-12-22
- Assignees: Peking University Founder Group Co., Ltd.; Peking University; Beijing Founder Electronics Co., Ltd.; Peking University Founder R & D Center
- Applicants: Song Wang; Rufeng Liang; Dan Zhang; Hong'e Sun; Jianwu Yang; Xinli Wu
- Inventors: Song Wang; Rufeng Liang; Dan Zhang; Hong'e Sun; Jianwu Yang; Xinli Wu
- Web page: https://patentgrep.com/patent/US9519718B2

## CPC

- G06F 16/9566: {URL specific, e.g. using aliases, detecting broken or misspelled links}
- G06F 16/313: {Selection or weighting of terms for indexing}
- G06F 16/3334: {Selection or weighting of terms from queries, including natural language queries}
- G06F 16/3344: {using natural language analysis}
- G06F 16/951: Indexing; Web crawling techniques
- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present application provides web information detecting method and system. The method according to the present application comprises: pre-extracting keywords from a web page; storing a corresponding relationship between the extracted keywords and a URL of the web page in a database; obtaining a source file of a web page to be detected; searching the database for keywords that have the same URL as that of the web page to be detected; comparing the searched keywords to the source file information of the web page to be detected; and determining the existence of information of the web page to be detected according to a matching degree. The present application increases the accuracy of web information detection.

## Claims

### Claim 1

A computer implemented method for detecting web information comprising:
pre-extracting keywords from a web page;
storing a corresponding relationship between the extracted keywords and a URL of the web page in a database;
obtaining source file information of a web page to be detected;
searching the database for the keywords that have the same URL as that of the web page to be detected;
comparing the searched keywords to the obtained source file information of the web page; and
determining that said web page exists according to a matching degree P;
wherein the step of pre-extracting comprises:
(1) obtaining source file information of the web page,
(2) extracting a main body text from the source file information, and
(3) extracting the keywords from the main body text; and
wherein step (1) comprises:
emulating a HTTP request for returning an information status code, wherein if the code is not 200 or if something unusual occurs, the information of the web page will be deleted, otherwise, step (1) further comprises:
obtaining a bit array and a coding format of the source file, and utilizing the obtained coding format to convert the obtained bit array of the source file into a character formatted source file.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the step (1) further comprises:
extracting a title from the information of the source file, and
segmenting the title;
wherein if there is no title or the title cannot be segmented, then the title will be set as empty.

### Claim 3 (depends on 2)

The method according to claim 2, wherein before extracting the main body text from the information of the source file, the method further comprises:
filtering the source file information to keep a partitionable type tag and the information in the kept tag.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the step (2) further comprises:
(a) dividing the filtered source file information into text blocks based on the partitionable type tags, and storing contents of all the text blocks as well as distance between each block and its next adjacent text block;
(b) choosing the text block with most characters as a base text block;
(c) determining a upper limit block and a lower limit block via a relationship between a pre-set threshold and a distance ratio of characters number in the above and below text blocks to the base text block, wherein contents between the upper limit block and the lower limit block are taken as a main body text.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the step (a) further comprises:
storing content of each text block as well as distance between each block and its next adjacent text block via a list;
wherein the stored data has two attributes, i.e. the character information within the text block and the distance to the next adjacent text block.

### Claim 6 (depends on 4)

The method according to claim 4, wherein the step (a) further comprises:
storing content of each text block as well as distance between each block and the next adjacent text block via a bit array;
wherein the character information within the text blocks are scattered into the bit array, and the distance between two adjacent text blocks is identified by the number of blanks between the stored locations of the two text blocks in the arrays.

### Claim 7 (depends on 4)

The method according to claims 4, wherein the step (a) further comprises a step of calculating the distance between a text block and its adjacent text block, wherein the step of calculating further includes:
assuming that two adjacent text blocks are A1 and A2, there are partitionable type tags B1. . . Bn, the weights of the tags are WB1. . . WBn, and the number of the partitionable type tags between A1 and A2 are as nB1. . . nBn, respectively; and
calculating the distance dA1A2 between A1 and A2 as
dA1A2=nB1×WB1+nB2×WB2+. . . +nBn×WBn.

### Claim 8 (depends on 4)

The method according to claim 4, wherein the step (C) further includes:
setting a text block A as the base block, wherein the text block A has a characters,
setting an adjacent text block above the base block as text block A1, wherein the text block A1 has a1 characters,
setting an adjacent text block below the base block as text block A2, wherein the text block A2 has a2 characters,
setting a distance d1 between the text block A1 and the text block A, setting a distance d2 between the text block A and the text block A2, and setting a threshold of M;
if a1/d1=M1≧M and a2/d2=M2≧M, taking a average Mavg of M1 and M2 as a measured scale for calculating the upper and lower limit;
if only one of M1 and M2 is larger or equal to the threshold M, taking an average of said one and M to obtain a new Mavg as the measured scale for calculating the upper and lower limit,
else, taking the text block as the main body text sample;
after the Mavg is calculated, the step (C) further includes reorganizing the text block A, and the step of reorganizing further includes:
if a1/d1=M1≧M, and a2/d2=M2≧M, then taking A′=A1+A+A2;
taking the text block A1 as a new base block, and searching from A1 in an upward direction;
comparing the Mavg to a ratio of the number of characters in a upper adjacent text block As1 to the distance ds1 between A1 and As1,
if the ratio is greater then Mavg, then incorporating the content of the text block As1 into the text block A, and then searching in an upward direction with As1 as the new base block until one of the text blocks does not satisfy the requirement, similarly, repeating the search in a down direction with A2 as the base block until one of the text blocks does not satisfy the requirement, and taking the last text block A as the main body text sample; and
if one of M1 and M2 is larger or equal to the threshold M, assuming M1≧M, then taking A′=A+A1, searching in the upward direction with text block A1 as the base block until one of the text blocks does not satisfy the requirement, and taking the last text block A′ as the main body text sample.

### Claim 9 (depends on 4)

The method according to claim 4, further comprising:
if the title is empty, taking the main body text sample in step (b) as the main body text;
otherwise,
(i) verifying the main body text sample in accordance with the title segments;
(ii) if the verification fails, setting the text block with most characters as the base text block, repeating the step(c) and proceeding with verification of step (i);
if the verification is success, taking the main body text sample as the main body text.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the step (i) for verifying the main body text sample further includes:
comparing the title segments to the main body text sample to obtain a matching degree P′;
determining whether the main body text sample is the main body text in accordance with the matching degree P′, if the matching degree P′ is larger or equal to a preset threshold M′, the sample is verified; otherwise, the verification fails.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the matching degree P′ is calculated by rule of:
P′=nw1×w1+nw2×w2+ . . . +nwn×wn;
wherein W1 . . . Wn are the title segments, and w1 . . . wn are weights of the title segments, nw1 . . . nwn are matching numbers of the title segments in the main body text.

### Claim 12 (depends on 2)

The method according claim 2, wherein the step (3) of extracting the keywords from the main body text further includes:
(1) extracting all sections of the main body text, and counting the characters number in each of the sections; and
(2) extracting an abstract from the main body text,
if there is no title, automatically taking the section with most characters in the main body text as the abstract,
if there is a title, segmenting the title, and calculating a matching degree P″ of the title and the section with most characters in the main body text by rule of
P″=n′w1×w1+n′w2×w2+ . . . +n′wn×wn
where,
w1 . . . wn represent as weights for the title segments respectively, they have same value and representation as their counter-parts in equation 1, n′w1 . . . n′wn represent the matching numbers of the title segments in the section that has most characters respectively,
if the matching degree P″ is greater then 0, then the section is used as an abstract, otherwise, verifying the section with second most characters, and so on; and
(3) segmenting the abstract, and extracting the keywords from the abstract,
if the title is empty, calculating the frequency of an abstract segment appears in the abstract, and extracting the keywords from the abstract in a descending order of appearance frequency,
if there is a title, calculating a matching degree P ′″ between the title segments and the abstract segments by rule of
P′″=n″w1×w1+n″w2×w2+. . . +n″wn×wn
wherein, w1 . . . wn represent weights for the title segments W1 . . . W2 respectively, n″w1 . . . n″wn represent the matching number of the title segments in the abstract segments respectively, the title segments is extracted based on a descending order of P′″,
calculating the appearance frequency of an abstract segment appears in the abstract, and extracting the keywords from the abstract in a descending order in terms of the appearance frequency,
deleting the segments that appears in both title and abstract segments, then extracting a few segments from the remaining segments as keywords.

### Claim 13 (depends on 1)

The method according to claim 1, wherein before comparing the keywords to the source file information of the web page to be detected, the method further includes filtering the source file information of the web page to be detected to remove useless information, the useless information includes a title.

### Claim 14

A system for detecting information on a web page, including:
a extracting device (11) configured to extract keywords from a web page;
a storing device (12) configured to store a corresponding relationship between the extracted keywords and a URL of the web page in a database;
an obtaining device (13) configured to obtain a source file of the web page; and
a comparing device (14) configured to search the database for the keywords that have the same URL as that of the web page, to compare the searched keywords to the source file information of the web page and to determine that the web page exists according to a matching degree P;
wherein the extracting device (11) is further configured to perform the steps of:
(1) obtaining source file information of the web page,
(2) extracting a main body text from the source file information, and
(3) extracting the keywords from the main body text; and
wherein step (1) comprises:
emulating a HTTP request for returning an information status code, wherein if the code is not 200 or if something unusual occurs, the information of the web page will be deleted, otherwise, step (1) further comprises:
obtaining a bit array and a coding format of the source file, and utilizing the obtained coding format to convert the obtained bit array of the source file into a character formatted source file.

### Claim 15 (depends on 14)

The system according to claim 14, further including a filtering device (15) configured to filter out useless information of the source file of the web page.