# US 2019/0171939 A1 · Method of processing image data in a connectionist network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171939 A1
- Document ID: US20190171939A1
- Status: Published application
- Kind: A1
- Application number: 16/202,688
- Filed: 2018-11-28
- Published: 2019-06-06
- Earliest claimed: 2017-12-05
- Assignees: Aptiv Technologies AG
- Applicants: Aptiv Technologies Limited
- Inventors: Weimeng Zhu; Jan Siegemund
- Web page: https://patentgrep.com/patent/US20190171939A1

## CPC

- G06V 20/582: {of traffic signs}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of processing image data in a connectionist network includes: determining, a plurality of offsets, each offset representing an individual location shift of an underlying one of the plurality of output picture elements, determining, from the plurality of offsets, a grid for sampling from the plurality of input picture elements, wherein the grid comprises a plurality of sampling locations, each sampling location being defined by means of a respective pair of one of the plurality of offsets and the underlying one of the plurality of output picture elements, sampling from the plurality of input picture elements in accordance with the grid, and transmitting, as output data for at least a subsequent one of the plurality of units of the connectionist network, a plurality of sampled picture elements resulting from the sampling, wherein the plurality of sampled picture elements form the plurality of output picture elements.

## Claims

### Claim 1

A method of processing image data in a connectionist network comprising a plurality of units (10, 38, 72, 74, 76, 79), each unit having a data input (12), a data output (14), and at least one processing parameter, wherein processing of input data by means of the unit is at least partially determined by the at least one processing parameter of the unit,
wherein the method implements at least a respective one (10) of the plurality of units, and wherein the method comprises:
receiving, as raw input data (58, 60, 62, 64) or input data processed by a preceding one (72) of the plurality of units, a plurality of input picture elements (24) representing an image acquired by means of an image sensor,
determining, for a plurality of output picture elements, a plurality of offsets (21), each offset (21) representing an individual location shift of an underlying one of the plurality of output picture elements,
determining, from the plurality of offsets (21), a grid (30) for sampling from the plurality of input picture elements (24), wherein the grid (30) comprises a plurality of sampling locations (32), each sampling location (32) being defined by means of a respective pair of one of the plurality of offsets (21) and the underlying one of the plurality of output picture elements,
sampling from the plurality of input picture elements (24) in accordance with the grid (30), and
transmitting, as output data (18) for at least a subsequent one (38, 74, 76, 78) of the plurality of units of the connectionist network, a plurality of sampled picture elements (33) resulting from the sampling, wherein the plurality of sampled picture elements (33) form the plurality of output picture elements.

### Claim 2 (depends on 1)

The method according to claim 1,
wherein the individual location shift represented by a respective one of the plurality of offsets (21) defines an arbitrary location relative to the underlying one of the plurality of output picture elements.

### Claim 3 (depends on 1)

The method according to claim 1,
wherein, for a respective one of the plurality of sampling locations (32) of the grid (30), sampling comprises interpolating a respective one (34, 36) of the plurality of output picture elements associated with the respective one of the plurality of sampling locations (32) when the respective one of the plurality of sampling locations (32) is offside any of the plurality of input picture elements (24).

### Claim 4 (depends on 1)

The method according to claim 1,
wherein at least some of the plurality of offsets (21) are spatially limited to a predefined threshold.

### Claim 5 (depends on 1)

The method according to claim 1,
wherein determining the plurality of offsets (21) is carried out by means of a localization connectionist network (20), in particular neural network, comprising one or more units, each unit having a data input, a data output, and at least one processing parameter, wherein processing of the input data by means of the one or more units is at least partially determined by the at least one processing parameter.

### Claim 6 (depends on 5)

The method according to claim 5,
wherein the one or more units of the localization connectionist network (20) are convolutional units, wherein a respective convolutional unit implements a convolution of at least some of the plurality of input picture elements (24) received by the respective convolutional unit with a kernel filter, and where-in the sampling does not comprise a convolution with a kernel filter.

### Claim 7 (depends on 5)

The method according to claim 5,
wherein the localization connectionist network (20) is trained together with the connectionist network by means of the feed-forward algorithm and the back-propagation algorithm, wherein training comprises modifying the at least one processing parameter of at least some of the plurality of units of the connectionist network and/or of the one or more units of the localization connectionist network (20).

### Claim 8 (depends on 7)

The method according to claim 7,
wherein during training of the localization connectionist network (20) and the connectionist network and if the unit (10) implemented by the method receives input data processed by a preceding one (72) of the plurality of units of the connectionist network, training data from the localization connectionist network (20) is selectively not used for modifying the at least one processing parameter of the preceding one (72) of the plurality of units of the connectionist network (20).

### Claim 9 (depends on 7)

The method according to claim 7,
the localization connectionist network (20) and the connectionist network are trained before using the connectionist network for a predefined purpose.

### Claim 10 (depends on 1)

The method according to claim 1,
wherein at least one of the plurality of units of the connectionist network is a convolutional unit, wherein a respective convolutional unit implements a convolution of at least some of the plurality of input picture elements (24) or output picture elements (33) received by the respective convolutional unit with a kernel filter.

### Claim 11 (depends on 1)

The method according to claim 1,
wherein the connectionist network implements a classifier for at least parts of the image represented by the plurality of input picture elements (24).

### Claim 12 (depends on 1)

The method according to claim 1,
wherein the connectionist network implements a classifier for traffic signs.

### Claim 13 (depends on 1)

The method according to claim 1,
wherein the output picture elements (33) are transmitted as output data to a plurality of subsequent units (74, 76, 78) of the connectionist network.

### Claim 14 (depends on 1)

The method according to claim 1,
wherein the sampling does not comprise a convolution of the plurality of in-put picture elements (24) with a kernel filter.

### Claim 15 (depends on 1)

A vehicle with a camera unit having an image sensor acquiring at least one image of an object in the vicinity of the vehicle,
the vehicle further comprising a processing unit configured to carry out the method according to claim 1.
