# US 10,872,273 B2 · System and method for batch-normalized recurrent highway networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,273 B2
- Document ID: US10872273B2
- Status: Granted
- Kind: B2
- Application number: 15/968,012
- Filed: 2018-05-01
- Published: 2018-11-08
- Granted: 2020-12-22
- Earliest claimed: 2017-05-02
- Assignees: Kodak Alaris Inc.
- Applicants: Kodak Alaris Inc.
- Inventors: Chi Zhang; Raymond Ptucha; Alexander Loui; Carl Salvaggio
- Web page: https://patentgrep.com/patent/US10872273B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments of the present disclosure relate to a recurrent framework based on Recurrent Highway Networks (RHNs) for sequence modeling using batch normalization. In certain embodiments, constraints within the RHNs are relaxed to reduce or avoid gradient vanishing or exploding by normalizing the current transition units in highway layers.

## Claims

### Claim 1

A computer-implemented method for image object recognition and captioning, comprising:
loading a pre-trained model into a multi-layer Recurrent Highway Network;
receiving a digital image input on which to perform object recognition, the digital image input containing one or more objects;
batch normalizing the digital image input by applying one or more non-linear transforms to the digital image input to generate transform outputs from the multi-layer Recurrent Highway Network; and
combining the transform outputs to generate a final output from the multi-layer Recurrent Highway Network, wherein the final output comprises a caption describing the one or more objects recognized in the digital image input.

### Claim 2 (depends on 1)

The method of claim 1 wherein the pre-trained model contains parameters for the computing network nodes.

### Claim 3 (depends on 1)

The method of claim 1 wherein the multi-layer neural network is a batch-normalized Recurrent Highway Network.

### Claim 4 (depends on 3)

The method of claim 3 wherein the constraints within the Recurrent Highway Network are relaxed to reduce or avoid gradient vanishing or exploding by normalizing the current transition units in highway layers.

### Claim 5 (depends on 4)

The method of claim 4, wherein a gradient norm clipping strategy is adopted to improve language semantics of the caption.

### Claim 6

A computer-implemented method for image object recognition and captioning, comprising:
receiving a digital image input on which to perform object recognition, the digital image input containing one or more objects;
batch normalizing the digital image input by applying one or more non-linear transforms to the digital image input to generate transform outputs from a multi-layer Recurrent Highway Network, wherein batch normalizing occurs at one or more highway layers of the multi-layer Recurrent Highway Network, wherein the one or more highway layers comprise a transform gate and a carry gate, and wherein the transform gate is coupled to the carry gate; and
combining the transform outputs to generate a final output, wherein the final output comprises a caption describing the one or more objects in the digital image input.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the digital image input is an output of a prior processing loop in the recurrent highway network.

### Claim 8 (depends on 6)

The computer-implemented method of claim 6, wherein batch normalizing the digital image input comprises extracting a mean and a variance across each channel and spatial location.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein batch normalizing the digital image input comprises subtracting the mean of the digital image input and dividing by the variance of the digital image input.