# US 2021/0081677 A1 · Unsupervised Video Object Segmentation and Image Object Co-Segmentation Using Attentive Graph Neural Network Architectures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081677 A1
- Document ID: US20210081677A1
- Status: Published application
- Kind: A1
- Application number: 16/574,864
- Filed: 2019-09-18
- Published: 2021-03-18
- Earliest claimed: 2019-09-18
- Applicants: Inception Institute of Artificial Intelligence, Ltd.
- Inventors: Wenguan Wang; Jianbing Shen; Xiankai Lu; Ling Shao
- Web page: https://patentgrep.com/patent/US20210081677A1

## CPC

- G06F 16/783: using metadata automatically derived from the content
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06F 18/29: {Graphical models, e.g. Bayesian networks}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 7/174: involving the use of two or more images
- G06T 7/215: Motion-based segmentation
- G06V 10/426: Graphical representations
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 10/84: using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06T 2207/20072: Graph-based image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

This disclosure relates to improved techniques for performing image segmentation functions using neural network architectures. The neural network architecture can include an attentive graph neural network (AGNN) that facilitates performance of unsupervised video object segmentation (UVOS) functions and image object co-segmentation (IOCS) functions. The AGNN can generate a graph that utilizes nodes to represent images (e.g., video frames) and edges to represent relations between the images. A message passing function can propagate messages among the nodes to capture high-order relationship information among the images, thus providing a more global view of the video or image content. The high-order relationship information can be utilized to more accurately perform UVOS and/or IOCS functions.

## Claims

### Claim 1

A system comprising:
one or more computing devices comprising one or more processors and one or more non-transitory storage devices for storing instructions, wherein execution of the instructions by the one or more processors causes the one or more computing devices to:
receive, at an attentive graph neural network architecture, a plurality of images;
execute, using the attentive graph neural network architecture, one or more segmentation functions on the images, at least in part, by:
extracting, using a feature extraction component associated with the attentive graph neural network architecture, node embeddings from the images;
creating a graph that comprises a plurality of nodes that are interconnected by a plurality of edges, wherein each node of the graph is associated with one of the node embeddings extracted using the feature extraction component;
determining, using one or more attention functions associated with the attentive graph neural network architecture, edge embeddings that capture relationship information associated with the node embeddings, wherein each edge of the graph is associated with one of the edge embeddings; and
executing, using the attentive graph neural network architecture, a message passing function that updates the node embeddings for each of the nodes, wherein the message passing function enables each node to update its corresponding node embedding, at least in part, using the relationship information of the edge embeddings corresponding to the edges that are connected to the node; and
generate segmentation results based, at least in part, on the updated node embeddings associated with the nodes.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more segmentation functions executed by the attentive graph neural network architecture include an unsupervised video object segmentation function.

### Claim 3 (depends on 2)

The system of claim 2, wherein:
the plurality of images correspond to frames of a video;
the unsupervised video object segmentation function is configured to generate segmentation results that identify or segment one or more objects included in at least a portion of the frames associated with the video.

### Claim 4 (depends on 1)

The system of claim 1, wherein the one or more segmentation functions executed by the attentive graph neural network architecture include an image object co-segmentation function.

### Claim 5 (depends on 4)

The system of claim 4, wherein
at least one of the images include common objects belonging to a semantic class; and
the object co-segmentation function is configured to jointly identify or segment the common objects included in the semantic class.

### Claim 6 (depends on 1)

The system of claim 1, wherein:
the graph is a fully-connected graph;
at least a portion of the edges are associated with line-edge embeddings that are obtained using an inter-node attention function; and
the line-edge embeddings capture pair-wise relationship information for node pairs included in the fully-connected graph.

### Claim 7 (depends on 6)

The system of claim 6, wherein:
at least a portion of the edges of the graph are associated with loop-edge embeddings that are obtained using an intra-node attention function; and
the loop-edge embeddings capture internal relationship information within the nodes of the fully-connected graph.

### Claim 8 (depends on 7)

The system of claim 7, wherein the message passing function updates the node embeddings for each of the nodes, at least in part, using the pair-wise relationship associated with the line-edge embeddings and the internal relationship information associated with the loop-edge embeddings.

### Claim 9 (depends on 1)

The system of claim 1, wherein the message passing function is configured to filter out information from noisy or irrelevant images included in the plurality of images.

### Claim 10 (depends on 1)

The system of claim 1, wherein the attentive graph neural network architecture is stored on an image capturing device or is configured to perform post-processing operations on images that are generated by an image capturing device.

### Claim 11

A method comprising:
receiving, at an attentive graph neural network architecture, a plurality of images;
executing, using the attentive graph neural network architecture, one or more segmentation functions on the images, at least in part, by:
extracting, using a feature extraction component associated with the attentive graph neural network architecture, node embeddings from the images;
creating a graph that comprises a plurality of nodes that are interconnected by a plurality of edges, wherein each node of the graph is associated with one of the node embeddings extracted using the feature extraction component;
determining, using one or more attention functions associated with the attentive graph neural network architecture, edge embeddings that capture relationship information associated with the node embeddings, wherein each edge of the graph is associated with one of the edge embeddings; and
executing, using the attentive graph neural network architecture, a message passing function that updates the node embeddings for each of the nodes, wherein the message passing function enables each node to update its corresponding node embedding, at least in part, using the relationship information of the edge embeddings corresponding to the edges that are connected to the node; and
generating segmentation results based, at least in part, on the updated node embeddings associated with the nodes.

### Claim 12 (depends on 11)

The method of claim 11, wherein the one or more segmentation functions executed by the attentive graph neural network architecture include an unsupervised video object segmentation function.

### Claim 13 (depends on 12)

The method of claim 12, wherein:
the plurality of images correspond to frames of a video;
the unsupervised video object segmentation function is configured to generate segmentation results that identify or segment one or more objects included in at least a portion of the frames associated with the video.

### Claim 14 (depends on 11)

The method of claim 11, wherein the one or more segmentation functions executed by the attentive graph neural network architecture include an image object co-segmentation function.

### Claim 15 (depends on 14)

The method of claim 14, wherein
at least one of the images include common objects belonging to a semantic class; and
the object co-segmentation function is configured to jointly identify or segment the common objects included in the semantic class.

### Claim 16 (depends on 11)

The method of claim 11, wherein:
the graph is a fully-connected graph;
at least a portion of the edges are associated with line-edge embeddings that are obtained using an inter-node attention function; and
the line-edge embeddings capture pair-wise relationship information for node pairs included in the fully-connected graph.

### Claim 17 (depends on 16)

The method of claim 16, wherein:
at least a portion of the edges of the graph are associated with loop-edge embeddings that are obtained using an intra-node attention function; and
the loop-edge embeddings capture internal relationship information within the nodes of the fully-connected graph.

### Claim 18 (depends on 17)

The method of claim 17, wherein the message passing function updates the node embeddings for each of the nodes, at least in part, using the pair-wise relationship associated with the line-edge embeddings and the internal relationship information associated with the loop-edge embeddings.

### Claim 19 (depends on 11)

The method of claim 11, wherein the message passing function is configured to filter out information from noisy or irrelevant images included in the plurality of images.

### Claim 20

A computer program product comprising a non-transitory computer-readable medium including instructions for causing a computer to:
receive, at an attentive graph neural network architecture, a plurality of images;
execute, using the attentive graph neural network architecture, one or more segmentation functions on the images, at least in part, by:
extracting, using a feature extraction component associated with the attentive graph neural network architecture, node embeddings from the images;
creating a graph that comprises a plurality of nodes that are interconnected by a plurality of edges, wherein each node of the graph is associated with one of the node embeddings extracted using the feature extraction component;
determining, using one or more attention functions associated with the attentive graph neural network architecture, edge embeddings that capture relationship information associated with the node embeddings, wherein each edge of the graph is associated with one of the edge embeddings; and
executing, using the attentive graph neural network architecture, a message passing function that updates the node embeddings for each of the nodes, wherein the message passing function enables each node to update its corresponding node embedding, at least in part, using the relationship information of the edge embeddings corresponding to the edges that are connected to the node; and
generate segmentation results based, at least in part, on the updated node embeddings associated with the nodes.
