# US 2025/0061736 A1 · Note generating method and related device thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0061736 A1
- Document ID: US20250061736A1
- Status: Published application
- Kind: A1
- Application number: 18/723,870
- Filed: 2022-12-26
- Published: 2025-02-20
- Earliest claimed: 2021-12-28
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Zhenyi He; Zhirong Xue
- Web page: https://patentgrep.com/patent/US20250061736A1

## CPC

- G06V 30/147: {Determination of region of interest}
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154)
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 30/146: Aligning or centring of the image pick-up or image-field
- G06V 30/19: Recognition using electronic means
- G06V 30/262: using context analysis, e.g. lexical, syntactic or semantic context
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In a note generating method, a terminal device obtains a target text area in a first image frame, wherein the target text area is a text area being read by a user. The terminal device converts a first drawn line in the target text area into a first detection area, which is used to identify a text area marked by the first drawn line. The terminal device then identifies a text area in the first detection area to obtain a user note.

## Claims

### Claim 1

A note generating method, wherein the method comprises:
obtaining a target text area in a first image frame, wherein the target text area is a to-be-identified text area;
converting a first drawn line in the target text area into a first detection area, wherein the first detection area is used to identify a text area marked by the first drawn line; and
identifying a text area in the first detection area to obtain a user note.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the converting a first drawn line in the target text area into a first detection area comprises:
creating a plurality of first rectangles that overlap with the first drawn line in the target text area, wherein the plurality of first rectangles are sequentially stacked;
creating a second drawn line in a first rectangle with a largest overlapping degree, wherein the second drawn line is parallel to a long side of the first rectangle with the largest overlapping degree; and
creating a second rectangle based on the second drawn line, wherein the second rectangle is used as the first detection area, the second drawn line is located in the second rectangle, the second drawn line is parallel to a long side of the second rectangle, and a length of a short side of the second rectangle is greater than a row height of the target text area.

### Claim 3 (depends on 2)

The method according to claim 2, wherein after the creating a second rectangle based on the second drawn line, the method further comprises:
dividing the second rectangle into a plurality of sub-rectangles; and
removing, from the plurality of sub-rectangles, a sub-rectangle whose pixel proportion is less than a preset first threshold, and using a third rectangle formed by remaining sub-rectangles as the first detection area.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the obtaining a target text area in a first image frame comprises:
when state information about text areas in the first image frame is different from state information about text areas in a second image frame, determining the target text area in the first image frame based on the state information about the text areas in the first image frame, wherein the second image frame is an image frame previous to the first image frame.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the state information about the text areas comprises at least one of the following: a quantity of the text areas, sizes of the text areas, angles of the text areas, and locations of the text areas.

### Claim 6 (depends on 4)

The method according to claim 4, wherein the determining the target text area in the first image frame based on the state information about the text areas in the first image frame comprises:
when there is a human body area of a user in the first image frame, comparing a quantity of text areas in the first image frame with a quantity of text areas in the second image frame, to detect whether a new text area exists in the first image frame; and
when a new text area exists in the first image frame, determining the new text area as the target text area; or
when there is no new text area in the first image frame, determining a text area associated with the human body area as the target text area; or
when there is no human body area of a user in the first image frame, determining a text area with a largest semantic size as the target text area, wherein a semantic size of a text area is a ratio of a size of the text area to a semantic distance of the text area, and the semantic distance of the text area is a distance between the text area and a central point of the first image frame.

### Claim 7 (depends on 1)

The method according to claim 1, wherein a second detection area further exists in the target text area, the second detection area is obtained by converting a third drawn line in a third image frame, the third image frame is previous to the first image frame, and a plurality of image frames exist between the third image frame and the first image frame; and the identifying a text area in the first detection area to obtain a user note comprises:
when a distance between the text area in the first detection area and a text area in the second detection area is greater than or equal to a preset second threshold, respectively identifying the text area in the first detection area and the text area in the second detection area to obtain two user notes; or
when a distance between the text area in the first detection area and a text area in the second detection area is less than a preset second threshold, merging the first detection area and the second detection area into a third detection area, and identifying a text area in the third detection area to obtain the user note.

### Claim 8 (depends on 7)

The method according to claim 7, wherein after the respectively identifying the text area in the first detection area and the text area in the second detection area to obtain two user notes, the method further comprises:
merging the two user notes to obtain a new user note, wherein the two user notes are located in a same paragraph, and the new user note comprises other texts other than the two user notes in the paragraph and the two user notes that are highlighted.

### Claim 9

(canceled)

### Claim 10 (depends on 1)

The method according to claim 1, wherein a target symbol exists in the target text area, and after the identifying a text area in the first detection area to obtain a user note, the method further comprises:
when the target symbol is comprised in a preset symbol set, adding the user note to a user note set corresponding to the target symbol; or
when the target symbol is not comprised in the symbol set, adding the target symbol to the symbol set, creating a user note set corresponding to the target symbol, and then adding the user note to the user note set corresponding to the target symbol.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the first detection area is a first color block, and the first color block is used to cover the text area marked by the first drawn line, and
wherein a format of the user note is determined based on an instruction entered by the user, and the format of the user note comprises at least one of the following: a font of the user note, a color of the user note, a thickness of the user note, a location of the user note, and a paragraph identifier of the user note.

### Claim at position 12

12-13. (canceled)

### Claim 14

A note generating apparatus, wherein the apparatus comprises:
an obtaining module, configured to obtain a target text area in a first image frame, wherein the target text area is a to-be-identified text area, and the first image frame originates from media information;
a conversion module, configured to convert a first drawn line in the target text area into a first detection area, wherein the first detection area is used to identify a text area marked by the first drawn line; and
an identification module, configured to identify a text area in the first detection area to obtain a user note.

### Claim 15 (depends on 14)

The apparatus according to claim 14, wherein the conversion module is configured to:
create a plurality of first rectangles that overlap with the first drawn line in the target text area, wherein the plurality of first rectangles are sequentially stacked;
create a second drawn line in a first rectangle with a largest overlapping degree, wherein the second drawn line is parallel to a long side of the first rectangle with the largest overlapping degree; and
create a second rectangle based on the second drawn line, wherein the second rectangle is used as the first detection area, the second drawn line is located in the second rectangle, the second drawn line is parallel to a long side of the second rectangle, and a length of a short side of the second rectangle is greater than a row height of the target text area.

### Claim 16 (depends on 15)

The apparatus according to claim 15, wherein the apparatus further comprises an optimization unit, configured to:
divide the second rectangle into a plurality of sub-rectangles; and
remove, from the plurality of sub-rectangles, a sub-rectangle whose pixel proportion is less than a preset first threshold, and use a third rectangle formed by remaining sub-rectangles as the first detection area.

### Claim 17 (depends on 14)

The apparatus according to claim 14, wherein the obtaining module is configured to: when state information about text areas in the first image frame is different from state information about text areas in a second image frame, determine the target text area in the first image frame based on the state information about the text areas in the first image frame; and the second image frame is an image frame previous to the first image frame.

### Claim 18 (depends on 17)

The apparatus according to claim 17, wherein the state information about the text areas comprises at least one of the following: a quantity of the text areas, sizes of the text areas, angles of the text areas, and locations of the text areas.

### Claim 19 (depends on 17)

The apparatus according to claim 17, wherein the obtaining module is configured to:
when there is a human body area of a user in the first image frame, compare a quantity of text areas in the first image frame with a quantity of text areas in the second image frame, to detect whether a new text area exists in the first image frame; and
when a new text area exists in the first image frame, determine the new text area as the target text area; or
when there is no new text area in the first image frame, determine a text area associated with the human body area as the target text area; or
when there is no human body area of a user in the first image frame, determine a text area with a largest semantic size as the target text area, wherein a semantic size of a text area is a ratio of a size of the text area to a semantic distance of the text area, and the semantic distance of the text area is a distance between the text area and a central point of the first image frame.

### Claim 20 (depends on 17)

The apparatus according to claim 17, wherein a second detection area further exists in the target text area, the second detection area is obtained by converting a third drawn line in a third image frame, the third image frame is previous to the first image frame, and a plurality of image frames exist between the third image frame and the first image frame; and the identification module is configured to:
when a distance between a text area in the first detection area and a text area in the second detection area is greater than or equal to a preset second threshold, respectively identify the text area in the first detection area and the text area in the second detection area to obtain two user notes; or
when a distance between the text area in the first detection area and a text area in the second detection area is less than a preset second threshold, merge the first detection area and the second detection area into a third detection area, and identify a text area in the third detection area to obtain the user note.

### Claim 21 (depends on 20)

The apparatus according to claim 20, wherein the apparatus further comprises a merging module, configured to merge the two user notes to obtain a new user note; the two user notes are located in a same paragraph; and the new user note comprises other texts other than the two user notes in the paragraph and the two user notes that are highlighted.

### Claim 22

(canceled)

### Claim 23 (depends on 14)

The apparatus according to claim 14, wherein a target symbol exists in the target text area, and the apparatus further comprises a classification module, configured to:
when the target symbol is comprised in a preset symbol set, add the user note to a user note set corresponding to the target symbol; or
when the target symbol is not comprised in the symbol set, add the target symbol to the symbol set, create a user note set corresponding to the target symbol, and then add the user note to the user note set corresponding to the target symbol.

### Claim 24 (depends on 14)

The apparatus according to claim 14, wherein the first detection area is a first color block, and the first color block is used to cover the text area marked by the first drawn line, and
wherein a format of the user note is determined based on an instruction entered by the user, and the format of the user note comprises at least one of the following: a font of the user note, a color of the user note, a thickness of the user note, a location of the user note, and a paragraph identifier of the user note.

### Claim at position 24

25-29. (canceled)
