# US 2021/0173395 A1 · Formally safe symbolic reinforcement learning on visual inputs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0173395 A1
- Document ID: US20210173395A1
- Status: Published application
- Kind: A1
- Application number: 16/709,633
- Filed: 2019-12-10
- Published: 2021-06-10
- Earliest claimed: 2019-12-10
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Subhro Das; Nathan Fulton; Nathan Hunt; Trong Nghia Hoang
- Web page: https://patentgrep.com/patent/US20210173395A1

## CPC

- G06N 3/08: Learning methods
- G05D 1/0055: {with safety arrangements}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 20/00: Machine learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for training control software to reinforce safety constraints using visual inputs includes performing template matching for each object in an image of a reinforcement learning (RL) agent's action space using a visual template for each object wherein each object in the RL agent's action space is detected, mapping each detected object to a set of planar coordinates for each object in the RL agent's action space, determining a set of safe actions for the RL agent by applying a safety specification for the RL agent's action space to the set of variables for coordinates for each object in the RL agent's action space, outputting the set of safe actions to the RL agent for a current state of a RL procedure, and preventing the RL agent from executing an action that is unsafe, before the RL agent takes an action.

## Claims

### Claim 1

A method for training control software to reinforce safety constraints using visual inputs, said method comprising the steps of:
performing template matching for each object in an image of a reinforcement learning (RL) agent's action space using a visual template for each object in the image wherein each object in the image of the RL agent's action space is detected;
mapping each detected object to a set of planar coordinates for each object in the image of the RL agent's action space,
determining a set of safe actions for the RL agent by applying a safety specification for the RL agent's action space to the set of variables for coordinates for each object in the image of the RL agent's action space;
outputting the set of safe actions to the RL agent for a current state of a RL procedure; and
preventing the RL agent from executing an action that is unsafe, before the RL agent takes an action.

### Claim 2 (depends on 1)

The method of claim 1, wherein performing template matching for each object in the image of the R L agent's action space comprises using the visual template for each object to place bounding boxes about each object in the image of the RL agent's action space.

### Claim 3 (depends on 2)

The method of claim 2, wherein mapping each detected object to the set of planar coordinates for each object in the image of the RL agent's action space comprises mapping a centroid of each bounding box to the planar coordinates for each object in the image of the RL agent's action space.

### Claim 4 (depends on 1)

The method of claim 1, further comprising training template matching for each object by augmenting data in a single, labeled image of the RL agent's action space, removing all templates from the image, randomly sampling a plurality of templates, augmenting each of the plurality of templates, randomly placing each of the plurality of templates in the image using template masks, wherein a plurality of images are created, and training the template matching using the plurality of images.

### Claim 5 (depends on 4)

The method of claim 4, wherein augmenting data in the single, labeled image includes one or more of randomly adjusting the brightness of the image, cropping the image, or left-right flipping of the image, and wherein augmenting each of the plurality of templates comprises one or more of left-right flipping of the template, rotating the template, or dropping pixels from the template.

### Claim 6 (depends on 1)

The method of claim 1, wherein the RL agent is one of a self-controlled land-based motor vehicle, a self-controlled aerial motor vehicle, a self-controlled floating aquatic motor vehicle, or a self-controlled underwater motor vehicle.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
using differences between the set of planar coordinates for each object at a current RL step and the set of planar coordinates for each object at a previous RL step to infer potential actions of each object;
wherein the RL agent uses knowledge of its dynamics and the safety specification to construct constraints that ensure that the RL agent takes an action only if, after taking the action and then following a worst possible outcome of all previously observed state transitions, the safety specification is still satisfied one step into the future.

### Claim 8 (depends on 1)

The method of claim 1, wherein the safety specification comprises a plurality of safety constraints specified in a formal logic for each object in the image.

### Claim 9

A method for training control software to reinforce safety constraints in a reinforcement learning (RL) algorithm that runs the control software, said method comprising the steps of:
forming visual templates of one or more safety-relevant objects in an input image from sample observations of each said safety-relevant object;
determining coordinates for each safety-relevant object from the visual template of each safety-relevant objects in the input image;
enforcing safety constraints for a RL agent in the RL algorithm by using the coordinates for each safety-relevant object; and
preventing the RL agent from executing an action that is unsafe, before the RL agent takes an action.

### Claim 10 (depends on 9)

The method of claim 9, wherein determining coordinates for each safety-relevant object from the visual template of each safety-relevant objects in the input image comprises placing bounding boxes each safety-relevant object by using the visual template for each object; and
mapping a centroid of each bounding box to the coordinates for each safety-relevant object in the input image.

### Claim 11 (depends on 9)

The method of claim 9, wherein enforcing safety constraints for the RL agent in the RL algorithm comprises receiving a safety specification for the RL agent, determining safety constraints from the safety specification and the coordinates for each safety-relevant object, wherein each action of the RL agent is labeled as safe or unsafe, and preventing the RL agent from executing an action that is unsafe.

### Claim 12 (depends on 9)

The method of claim 9, wherein the RL agent is one of a self-controlled land-based motor vehicle, a self-controlled aerial motor vehicle, a self-controlled floating aquatic motor vehicle, or a self-controlled underwater motor vehicle.

### Claim 13

A non-transitory program storage device readable by a computer, tangibly embodying a program of instructions executed by the computer to perform the method steps for training control software to reinforce safety constraints using visual inputs, said method comprising the steps of:
performing template matching for each object in an image of a reinforcement learning (RL) agent's action space using a visual template for each object in the image wherein each object in the image of the RL agent's action space is detected;
mapping each detected object to a set of planar coordinates for each object in the image of the RL agent's action space;
determining a set of safe actions for the RL agent by applying a safety specification for the RL agent's action space to the set of variables for coordinates for each object in the image of the RL agent's action space;
outputting the set of safe actions to the RL agent for a current state of a RL procedure, and
preventing the RL agent from executing an action that is unsafe, before the RL agent takes an action.

### Claim 14 (depends on 13)

The computer readable program storage device of claim 13, wherein performing template matching for each object in the image of the RL agent's action space comprises using the visual template for each object to place bounding boxes about each object in the image of the RL agent's action space.

### Claim 15 (depends on 14)

The computer readable program storage device of claim 14, wherein mapping each detected object to the set of planar coordinates for each object in the image of the RL agent's action space comprises mapping a centroid of each bounding box to the planar coordinates for each object in the image of the RL agent's action space.

### Claim 16 (depends on 13)

The computer readable program storage device of claim 13, the method further comprising training template matching for each object by augmenting data in a single, labeled image of the RL agent's action space, removing all templates from the image, randomly sampling a plurality of templates, augmenting each of the plurality of templates, randomly placing each of the plurality of templates in the image using template masks, wherein a plurality of images are created, and training the template matching using the plurality of images.

### Claim 17 (depends on 16)

The computer readable program storage device of claim 16, wherein augmenting data in the single, labeled image includes one or more of randomly adjusting the brightness of the image, cropping the image, or left-right flipping of the image, and wherein augmenting each of the plurality of templates comprises one or more of left-right flipping of the template, rotating the template, or dropping pixels from the template.

### Claim 18 (depends on 13)

The computer readable program storage device of claim 13, wherein the RL agent is one of a self-controlled land-based motor vehicle, a self-controlled aerial motor vehicle, a self-controlled floating aquatic motor vehicle, or a self-controlled underwater motor vehicle.

### Claim 19 (depends on 13)

The computer readable program storage device of claim 13, the method further comprising:
using differences between the set of planar coordinates for each object at a current RL step and the set of planar coordinates for each object at a previous RL step to infer potential actions of each object;
wherein the RL agent uses knowledge of its dynamics and the safety specification to construct constraints that ensure that the RL agent takes an action only if, after taking the action and then following a worst possible outcome of all previously observed state transitions, the safety specification is still satisfied one step into the future.

### Claim 20 (depends on 13)

The computer readable program storage device of claim 13, wherein the safety specification comprises a plurality of safety constraints specified in a formal logic for each object in the image.
