# US 2023/0267754 A1 · Automated Content Analysis and Annotation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267754 A1
- Document ID: US20230267754A1
- Status: Published application
- Kind: A1
- Application number: 17/675,953
- Filed: 2022-02-18
- Published: 2023-08-24
- Earliest claimed: 2022-02-18
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: Miquel Angel Farre Guiu; Monica Alfaro Vendrell; Marc Junyet Martin; Francesc Josep Guitart Bravo; Albert Aparicio Isarn; Pablo Pernias; Steven S. Shapiro; Anthony M. Accardo
- Web page: https://patentgrep.com/patent/US20230267754A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G06V 10/7753: {Incorporation of unlabelled data, e.g. multiple instance learning [MIL]}
- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/42: based on the type of document
- G06V 2201/131: Book
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system includes a computing platform having processing hardware, and a systems memory storing a software code. The processing hardware is configured to execute the software code to receive content including an image having multiple image regions, determine boundaries of each of the image regions to identify multiple bounded image regions, identify, within each of the bounded image regions, one or more local features and one or more global features, and identify, within each of the hounded image regions, another one or more local features based on a comparison with corresponding local features identified in each of one or more other bounded image regions. The processing hardware is further configured to execute the software code to annotate each of the bounded image regions using its respective one or more local features, its other one or more local features, and its one or more global features, to provide annotated content.

## Claims

### Claim 1

A system comprising:
a computing platform having a processing hardware and a system memory storing a software code;
the processing hardware configured to execute the software code to:
receive content including an image having a plurality of image regions;
determine a respective boundary of each of the plurality of image regions to identify a plurality of bounded image regions;
identify, within each of the plurality of bounded image regions, respective one or more local features and respective one or more global features;
identify, within each of the plurality of bounded image regions, another respective one or more local features based on a comparison with corresponding one or more local features identified in each of one or more other bounded image regions; and
annotate each of the plurality of bounded image regions using the respective one or more local features, the another respective one or more local features, and the respective one or more global features to provide an annotated content.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processing hardware is further configured to execute the software code to:
determine the respective boundary of each of the plurality of image regions using a first trained machine learning (ML) model of a plurality of trained ML models;
identify, within each of the plurality of bounded image regions, the respective one or more local features using one or more second trained ML model(s) of the plurality of trained ML models;
identify, within each of the plurality of bounded image regions, the respective one or more global features using one or more third trained ML model(s) of the plurality of trained ML models; and
identify, within each of the plurality bounded image regions, the another respective one or more local features using a fourth trained ML model of the plurality of trained ML models.

### Claim 3 (depends on 1)

The system of claim 1, wherein the content includes at least one sequence of bounded image regions sharing a relationship, and wherein, before annotating each of the plurality of bounded image regions included in the at least one sequence, the processing hardware is further configured to execute the software code to:
confirm or modify, using the relationship, for each of the bounded images regions of the at least one sequence of bounded image regions sharing the relationship, the respective one or more local features, the another respective one or more local features, and the respective one or more global features to provide confirmed or modified respective one or more local features and confirmed or modified respective one or more global features; and
annotate each of the bounded image regions of the at least one sequence of bounded image regions sharing the relationship further using the modified respective one or more local features and the modified respective one or more global features.

### Claim 4 (depends on 3)

The system of claim 3, wherein the processing hardware is further configured to execute the software code to:
identify a beginning and an end of each of the at least one sequence of bounded image regions sharing the relationship.

### Claim 5 (depends on 4)

The system of claim 4, wherein the processing hardware is further configured to execute the software code to:
identify the beginning and the end of each of the at least one sequence of bounded image regions sharing the relationship using a trained content segmentation machine learning (ML) model.

### Claim 6 (depends on 4)

The system of claim 4, wherein the processing hardware is further configured to execute the software code to:
identify the beginning and the end of each of the at least one sequence of bounded image regions sharing the relationship, in parallel with identifying the respective one or more local features and the respective one or more global features within each of the plurality of bounded image regions and identifying the another respective one or more local features within each of the plurality bounded image regions.

### Claim 7 (depends on 1)

The system of claim 1, wherein the annotated content comprises at least one of an annotated comic, an annotated graphic novel, or an annotated manga.

### Claim 8

A system comprising:
a computing platform having a processing hardware and a system memory storing a software code;
the processing hardware configured to execute the software code to:
receive content including an image having a plurality of image regions at least some of which form a sequence of image regions sharing a relationship;
determine a respective boundary of each of the plurality of mage regions to identify a plurality of bounded image regions;
identify, within each of the plurality of bounded image regions, respective one or more local features and respective one or more global features;
confirm or modify, using the relationship, for each of the bounded image regions corresponding respectively to the image regions sharing the relationship, the identified respective one or more local features and the identified respective one or more global features to provide confirmed or modified respective one or more local features and confirmed or modified respective one or more global features; and
annotate each of the bounded image regions corresponding respectively to the image regions sharing the relationship, using the confirmed or modified respective one or more local features and the confirmed or modified respective one or more global features to provide an annotated content.

### Claim 9 (depends on 8)

The system of claim 8, wherein the processing hardware is further configured to execute the software code to:
determine the respective boundary of each of the plurality of image regions using a first trained machine learning (ML) model of a plurality of trained ML models;
identify, within each of the plurality of bounded image regions, the respective one or more local features using one or more second trained ML model(s) of the plurality of trained ML models; and
identify, within each of the plurality of bounded image regions, the respective one or more global features using one or more third trained ML model(s) of the plurality of trained ML models.

### Claim 10 (depends on 8)

The system of claim 8, wherein the processing hardware is further configured to execute the software code to:
identify a beginning and an end of the sequence of image regions sharing the relationship.

### Claim 11 (depends on 10)

The system of claim 10, wherein the processing hardware is further configured to execute the software code to:
identify the beginning and the end of the sequence of image regions sharing the relationship using a trained content segmentation machine learning (ML) model.

### Claim 12 (depends on 10)

The system of claim 10, wherein the processing hardware is further configured to execute the software code to:
and identify the beginning and the end of the sequence of image regions sharing the relationship in parallel with identifying the respective one or more local features and the respective one or more global features within each of the plurality of bounded image regions.

### Claim 13 (depends on 8)

The system of claim 8, wherein the annotated content comprises at least one of an annotated comic, an annotated graphic novel, or an annotated manga.

### Claim 14

A method for use by a system including a computing platform having a processing hardware, and a system memory storing a software code, the method comprising:
receiving, by the software code executed by the processing hardware, content including an image having a plurality of image regions;
determining, by the software code executed by the processing hardware, a respective boundary of each of the plurality of image regions to identify a plurality of bounded image regions;
identifying, by the software code executed by the processing hardware within each of the plurality of bounded image regions, respective one or more local features and respective one or more global features;
identifying, by the software code executed by the processing hardware within each of the plurality of bounded image regions, another one or more local features based on a comparison with corresponding one or more local features identified in each of one or more other bounded image regions; and
annotating, by the software code executed by the processing hardware, each of the plurality of bounded image regions using the respective one or more local features, the another respective one or more local features, and the respective one or more global features to provide an annotated content.

### Claim 15 (depends on 14)

The method of claim 14 further comprising:
determining the respective boundary of each of the plurality of image regions, by the software code executed by the processing hardware, using a first trained machine learning (ML) model of a plurality of trained ML models;
identifying, by the software code executed by the processing hardware, the respective one or more local features within each of the plurality of bounded image regions using one or more second trained ML model(s) of the plurality of trained ML models;
identifying, by the software code executed by the processing hardware, the respective one or more global features within each of the plurality of bounded image regions using one or more third trained ML model(s) of the plurality of trained ML models; and
identifying, by the software code executed by the processing hardware, the another respective one or more local features within each of the plurality bounded image regions using a fourth trained ML model of the plurality of trained ML models.

### Claim 16 (depends on 14)

The method of claim 14, wherein the content includes at least one sequence of bounded image regions sharing a relationship, the method further comprising, before annotating each of the plurality of bounded image regions included in the at least one sequence:
confirming or modifying, for each of the bounded images regions of the at least one sequence of bounded image regions sharing the relationship, by the software code executed by the processing hardware and using the relationship, the respective one or more local features, the another respective one or more local features, and the respective one or more global features to provide confirmed or modified respective one or more local features and confirmed or modified respective one or more global features; and
annotating, by the software code executed by the processing hardware, each of the bounded image regions of the at least one sequence of bounded image regions sharing the relationship further using the modified respective one or more local features and the modified respective one or more global features.

### Claim 17 (depends on 16)

The method of claim 16, the method further comprising:
identifying, by the software code executed by the processing hardware, a beginning and an end of each of the at least one sequence of bounded image regions sharing the relationship.

### Claim 18 (depends on 16)

The method of claim 16 further comprising:
identifying the beginning and the end of each at least one sequence of bounded image regions sharing the relationship using a trained content segmentation machine learning (ML) model.

### Claim 19 (depends on 16)

The method of claim 16, further comprising:
identifying the beginning and the end of each of the at least one sequence of bounded image regions sharing the relationship, in parallel with identifying the respective one or more local features and the respective one or more global features within each of the plurality of hounded image regions and identifying the another respective one or more local features within each of the plurality hounded image regions.

### Claim 20 (depends on 14)

The method of claim 14, wherein the annotated content comprises at least one of an annotated comic, an annotated graphic novel, or an annotated manga.
