# US 2021/0334592 A1 · Reinforcement learning model for labeling spatial relationships between images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334592 A1
- Document ID: US20210334592A1
- Status: Published application
- Kind: A1
- Application number: 16/861,124
- Filed: 2020-04-28
- Published: 2021-10-28
- Earliest claimed: 2020-04-28
- Assignees: OMRON Corporation
- Applicants: OMRON Corporation
- Inventors: Tatsunori Taniai
- Web page: https://patentgrep.com/patent/US20210334592A1

## CPC

- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/092: Reinforcement learning
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/593: from stereo images
- G06V 10/7515: {Shifting the patterns to accommodate for positional errors}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A computing system is provided, including a processor configured to receive a labeling map for a first image. The labeling map may indicate a spatial relationship between a first region of interest included in the first image and a second region of interest included in a second image. At a trained reinforcement learning model, the processor may be further configured to generate an updated labeling map for the first image based on at least the labeling map, the first image, and the second image.

## Claims

### Claim 1

A computing system comprising:
a processor configured to:
receive a labeling map for a first image, wherein the labeling map indicates a spatial relationship between a first region of interest included in the first image and a second region of interest included in a second image; and
at a trained reinforcement learning model, generate an updated labeling map for the first image based on at least the labeling map, the first image, and the second image.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein:
the trained reinforcement learning model is trained using a plurality of training images; and
when training the trained reinforcement learning model, for each of the plurality of training images, the processor is configured to:
at an agent module, determine a candidate labeling update action for a candidate labeling map associated with the training image at least in part by applying a policy function with one or more agent parameters to the candidate labeling map, the training image, and an additional training image of the plurality of training images;
at an evaluation module, determine an evaluation metric value for the candidate labeling map;
at an environment module, determine:
an updated candidate labeling map based on at least the candidate labeling map and the candidate labeling update action; and
a reward value based on the candidate labeling update action and the evaluation metric value; and
at an updater module, modify the one or more agent parameters of the agent module based at least in part on the candidate labeling map, the candidate labeling update action, the updated candidate labeling map, and the reward value.

### Claim 3 (depends on 2)

The computing system of claim 2, wherein the candidate labeling map indicates a candidate spatial relationship between a training region of interest of the training image and an additional training region of interest of the additional training image.

### Claim 4 (depends on 2)

The computing system of claim 2, wherein, at the evaluation module, the processor is further configured to:
receive a manually generated labeling map; and
determine the evaluation metric value based at least in part on a difference between the candidate labeling map and the manually generated labeling map.

### Claim 5 (depends on 2)

The computing system of claim 2, wherein, at the updater module, the processor is further configured to:
over a plurality of parameter updating cycles corresponding to the plurality of training images, determine an expected cumulative reward value based on the respective reward values determined at the environment module in those parameter updating cycles; and
select respective parameter values of the one or more agent parameters that increase the expected cumulative reward value in a current parameter updating cycle.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein the labeling map is an optical flow map of a plurality of optical flow values between pixels that are respectively included in the first region of interest of the first image and the second region of interest of the second image.

### Claim 7 (depends on 1)

The computing system of claim 1, wherein the labeling map is a disparity map of a plurality of pixel location disparities between pixels that are respectively included in the first region of interest of the first image and the second region of interest of the second image.

### Claim 8 (depends on 1)

The computing system of claim 1, wherein the labeling map is a depth map of a plurality of spatial depth values of pixels that are respectively included in the first region of interest of the first image and the second region of interest of the second image.

### Claim 9 (depends on 1)

The computing system of claim 1, wherein the processor is further configured to generate a first feature map of the first region of interest and a second feature map of the second region of interest at one or more convolutional layers.

### Claim 10 (depends on 1)

The computing system of claim 1, wherein the first region of interest and the second region of interest each include a respective plurality of spatially contiguous pixels.

### Claim 11 (depends on 1)

The computing system of claim 1, wherein the processor is further configured to iteratively update the updated labeling map at the trained reinforcement learning model over a plurality of labeling map updating cycles.

### Claim 12 (depends on 1)

The computing system of claim 1, wherein the trained reinforcement learning model is a multi-agent reinforcement learning model.

### Claim 13

A method for use with a computing system, the method comprising:
receiving a labeling map for a first image, wherein the labeling map indicates a spatial relationship between a first region of interest included in the first image and a second region of interest included in a second image; and
at a trained reinforcement learning model, generating an updated labeling map for the first image based on at least the labeling map, the first image, and the second image.

### Claim 14 (depends on 13)

The method of claim 13, further comprising training the trained machine learning model with a plurality of training images at least in part by:
at an agent module, determining a candidate labeling update action for a candidate labeling map associated with the training image at least in part by applying a policy function with one or more agent parameters to the candidate labeling map, the training image, and an additional training image of the plurality of training images;
at an evaluation module, determining an evaluation metric value for the candidate labeling map;
at an environment module, determining:
an updated candidate labeling map based on at least the candidate labeling map and the candidate labeling update action; and
a reward value based on the candidate labeling update action and the evaluation metric value; and
at an updater module, modifying the one or more agent parameters of the agent module based at least in part on the candidate labeling map, the candidate labeling update action, the updated candidate labeling map, and the reward value.

### Claim 15 (depends on 14)

The method of claim 14, further comprising, at the evaluation module:
receiving a manually generated labeling map; and
determining the evaluation metric value based at least in part on a difference between the candidate labeling map and the manually generated labeling map.

### Claim 16 (depends on 14)

The method of claim 14, further comprising, at the updater module:
over a plurality of parameter updating cycles corresponding to the plurality of training images, determining an expected cumulative reward value based on the respective reward values determined at the environment module in those parameter updating cycles; and
selecting respective parameter values of the one or more agent parameters that increase the expected cumulative reward value in a current parameter updating cycle.

### Claim 17 (depends on 13)

The method of claim 13, wherein the labeling map is an optical flow map of a plurality of optical flow values between pixels that are respectively included in the first region of interest of the first image and the second region of interest of the second image.

### Claim 18 (depends on 13)

The method of claim 13, wherein the labeling map is a disparity map of a plurality of pixel location disparities between pixels that are respectively included in the first region of interest of the first image and the second region of interest of the second image.

### Claim 19 (depends on 13)

The method of claim 13, wherein the labeling map is a depth map of a plurality of spatial depth values of pixels that are respectively included in the first region of interest of the first image and the second region of interest of the second image.

### Claim 20

A computing system comprising:
one or more processors configured to:
generate a trained reinforcement learning model using a plurality of training images, at least in part by, for each of the training images:
at an agent module, determining a candidate labeling update action for a candidate labeling map associated with the training image at least in part by applying a policy function with one or more agent parameters to the candidate labeling map, the training image, and an additional training image of the plurality of training images;
at an evaluation module, determining an evaluation metric value for the candidate labeling map;
at an environment module, determining:
an updated candidate labeling map based on at least the candidate labeling map and the candidate labeling update action; and
a reward value based on the candidate labeling update action and the evaluation metric value; and
at an updater module, modifying the one or more agent parameters of the agent module based at least in part on the candidate labeling map, the candidate labeling update action, the updated candidate labeling map, and the reward value; and
at runtime:
receive a labeling map for a first image, wherein:
the labeling map indicates a spatial relationship between a first region of interest included in the first image and a second region of interest included in a second image; and
the labeling map is an optical flow map, a disparity map, or a depth map; and
at a trained reinforcement learning model, generate an updated labeling map for the first image based on at least the labeling map, the first image, and the second image.
