# US 11,443,133 B2 · Computer vision system for industrial equipment gauge digitization and alarms

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,133 B2
- Document ID: US11443133B2
- Status: Granted
- Kind: B2
- Application number: 16/709,035
- Filed: 2019-12-10
- Published: 2021-06-10
- Granted: 2022-09-13
- Earliest claimed: 2019-12-10
- Assignees: Ricoh Company, Ltd.
- Applicants: Edward L. Schwartz
- Inventors: Edward L. Schwartz
- Web page: https://patentgrep.com/patent/US11443133B2

## CPC

- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system for analog gauge monitoring uses a machine learning model for computer vision that is trained using synthetic training data generated based on one or a few images of the gauge being monitored and a geometric model describing the scale and the indicator of the gauge. In some embodiments, the synthetic training data is generated using an image model implemented as a generative adversarial network (GAN) type neural network and trained to modify an image of a given gauge such that the gauge face is preserved while the gauge indicator is added to or removed from the image of the given gauge for any given gauge.

## Claims

### Claim 1

A method for monitoring a gauge having a face including markings denoting a scale for a parameter value being measured by the gauge and an indicator indicating a reading of the parameter value over the scale by displacement of the indicator on the face, the method comprising:
receiving a set of training images of the gauge;
receiving data relating to annotations of the training images of the gauge, the annotations denoting the scale and the indicator on the face of the gauge;
generating a geometric model describing the gauge using geometric information derived from the data relating to the annotations;
providing an image model previously trained to modify an image of a given gauge such that a gauge face is preserved while a gauge indicator is added to or removed from the image of the given gauge for any given gauge, the previously trained image model being a neural network that is trained using a mask to distinguish the gauge indicator from the gauge face for any given gauge;
generating, using the geometric model and the previously trained image model, synthetic images of the gauge, the synthetic images having indicator at different positions relative to the scale, the synthetic images forming a training data set, wherein generating the synthetic images of the gauge comprises:
selecting an image patch from a respective training image of the gauge as input to the previously trained image model;
providing a reference image patch of the indicator of the gauge to the previously trained image model; and
applying the previously trained image model to erase the indicator from or draw the indicator onto the selected image patch;
training a machine learning model for computer vision using the training data set;
receiving operation images of the gauge during normal operation of the gauge; and
applying the trained machine learning model for computer vision to the operation images to predict readings of the gauge.

### Claim 2 (depends on 1)

The method of claim 1, wherein receiving the set of training images of the gauge comprises:
receiving one to five training images of the gauge as the set of training images, each training image being annotated to denote the scale and the indicator on the face of the gauge in that training image.

### Claim 3 (depends on 1)

The method of claim 1, wherein receiving data relating to annotations of the training images of the gauge comprises:
receiving data relating to the annotations denoting at least end points of the scale and at least end points of the indicator on the face of the gauge.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
receiving data relating to the annotations denoting a point on the scale between two endpoints.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating a geometric model describing the gauge using geometric information derived from the data relating to the annotations comprises:
generating the geometric model describing a geometric location of the scale on the face of the gauge and a geometric location and movement of the indicator on the face of the gauge.

### Claim 6 (depends on 1)

The method of claim 1, wherein the previously trained image model is a generative adversarial network (GAN).

### Claim 7 (depends on 6)

The method of claim 6, wherein selecting the image patch from the respective training image of the gauge includes selecting the image patch from the respective training image of the gauge using the geometric model.

### Claim 8 (depends on 7)

The method of claim 7, wherein generating, using the geometric model and the previously trained image model, the synthetic images of the gauge further comprises:
selecting a first image patch from a respective training image of the gauge as input to the previously trained image model, the first image patch containing the indicator of the gauge;
applying the previously trained image model to erase the indicator from the first image patch;
pasting the modified first image patch back onto the respective training image to generate a base image of the gauge without the indicator;
rotating the base image in a first direction by a first number of degrees;
selecting a second image patch from the base image of the gauge as input to the previously trained image model;
applying the previously trained image model to draw the indicator onto the second image patch;
pasting the modified second image patch back onto the base image to generate a synthetic image;
rotating the synthetic image in a second direction opposite the first direction by the first number of degrees;
providing the synthetic image as part of the training data set; and
repeating the rotating the base image using a different values of rotation degrees to providing the synthetic image to generate a plurality of synthetic images forming the training data set.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
training the machine learning model for computer vision using training data denoting a threshold value for the gauge; and
applying the trained machine learning model for computer vision to predict readings from operation images of the gauge as being above or below the threshold value.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
determining a time duration after receiving a first operation image; and
in response to the time duration exceeding a first time value, generating an alarm indicating that no subsequent operation image is being received for the time duration exceeding the first time value.

### Claim 11

A system for monitoring a gauge having a face including markings denoting a scale for a parameter value being measured by the gauge and an indicator indicating a reading of the parameter value over the scale by displacement of the indicator on the face, the system comprising:
a hardware processor; and
a memory coupled with the hardware processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
receive a set of training images of the gauge;
receive data relating to annotations of the training images of the gauge, the annotations denoting the scale and the indicator on the face of the gauge;
generate a geometric model describing the gauge using geometric information derived from the data relating to the annotations;
providing an image model previously trained to modify an image of a given gauge such that a gauge face is preserved while a gauge indicator is added to or removed from the image of the given gauge for any given gauge, the previously trained image model being a neural network that is trained using a mask to distinguish the gauge indicator from the gauge face for any given gauge;
generate, using the geometric model and the previously trained image model, synthetic images of the gauge, the synthetic images having indicator at different positions relative to the scale, the synthetic images forming a training data set, wherein to generate the synthetic images of the gauge, the instructions further cause the processor to:
select an image patch from a respective training image of the gauge as input to the previously trained image model;
provide a reference image patch of the indicator of the gauge to the previously trained image model; and
apply the previously trained image model to erase the indicator from or draw the indicator onto the selected image patch;
train a machine learning model for computer vision using the training data set;
receive operation images of the gauge during normal operation of the gauge; and
apply the trained machine learning model for computer vision to the operation images to predict readings of the gauge.

### Claim 12 (depends on 11)

The system of claim 11, wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:
receive one to five training images of the gauge as the set of training images, each training image being annotated to denote the scale and the indicator on the face of the gauge in that training image.

### Claim 13 (depends on 11)

The system of claim 11, wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:
receive data relating to the annotations denoting at least end points of the scale and at least end points of the indicator on the face of the gauge.

### Claim 14 (depends on 11)

The system of claim 11, wherein the previously trained image model is a generative adversarial network (GAN).

### Claim 15 (depends on 11)

The system of claim 11, wherein, to select the image patch from the respective training image of the gauge, the memory is further configured to provide the processor with instructions which when executed cause the processor to:
select the image patch from the respective training image of the gauge using the geometric model.

### Claim 16 (depends on 11)

The system of claim 11, wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:
train the machine learning model for computer vision using training data denoting a threshold value for the gauge; and
apply the trained machine learning model for computer vision to predict readings from operation images of the gauge as being above or below the threshold value.

### Claim 17

A non-transitory computer readable medium storing instructions which, when executed by a computer, cause the computer to:
receive a set of training images of a gauge;
receive data relating to annotations of the training images of the gauge, the annotations denoting a scale and an indicator on a face of the gauge;
generate a geometric model describing the gauge using geometric information derived from the data relating to the annotations;
provide an image model previously trained to modify an image of a given gauge such that a gauge face is preserved while a gauge indicator is added to or removed from the image of the given gauge for any given gauge, the previously trained image model being a neural network that is trained using a mask to distinguish the gauge indicator from the gauge face for any given gauge;
generate, using the geometric model and the previously trained image model, synthetic images of the gauge, the synthetic images having indicator at different positions relative to the scale, the synthetic images forming a training data set, wherein to generate the synthetic images of the gauge, the instructions further cause the computer to:
select an image patch from a respective training image of the gauge as input to the previously trained image model;
provide a reference image patch of the indicator of the gauge to the previously trained image model; and
apply the previously trained image model to erase the indicator from or draw the indicator onto the selected image patch;
train a machine learning model for computer vision using the training data set;
receive operation images of the gauge during normal operation of the gauge; and
apply the trained machine learning model for computer vision to the operation images to predict readings of the gauge.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein the instructions further cause the computer to:
receive one to five training images of the gauge as the set of training images, each training image being annotated to denote the scale and the indicator on the face of the gauge in that training image.

### Claim 19 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein the previously trained image model is a generative adversarial network (GAN).

### Claim 20 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein, to select the image patch from the respective training image of the gauge, the instructions further cause the computer to:
select the image patch from the respective training image of the gauge using the geometric model.