# US 10,769,422 B2 · Neural network-based recognition of trade workers present on industrial sites

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,422 B2
- Document ID: US10769422B2
- Status: Granted
- Kind: B2
- Application number: 16/135,942
- Filed: 2018-09-19
- Published: 2020-03-19
- Granted: 2020-09-08
- Earliest claimed: 2018-09-19
- Assignees: indus.ai Inc
- Applicants: indus.ai Inc
- Inventors: Lai Him Matthew Man; Mohammad Soltani; Ahmed Aly; Walid Aly
- Web page: https://patentgrep.com/patent/US10769422B2

## CPC

- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06V 10/245: {by locating a pattern; Special marks for positioning}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method and system for neural network-based recognition of trade workers present on industrial sites is presented. In an embodiment, a method comprises: using a computing device, receiving a plurality of digital images depicting a particular worker; using the computing device, based on the plurality of digital images, determining a plurality of key-point sets of the digital images, each of the plurality of key-point sets comprising location information of key points identified within a depiction of the particular worker in a particular digital image; using the computing device, based on the plurality of key-point sets, determining a plurality of trade-specific activities that appear to be performed by the particular worker; using the computing device, based on the plurality of trade-specific activities, determining a plurality of trade probabilities, each trade probability among the trade probabilities indicating a likelihood that the particular worker belongs to a particular trade from among a plurality of different trades.

## Claims

### Claim 1

A computer-implemented method comprising:
using a computing device, receiving a plurality of digital images depicting a particular worker;
using the computing device, based on the plurality of digital images, determining a plurality of key-point sets of the digital images, each of the plurality of key-point sets comprising location information of key points identified within a depiction of the particular worker in a particular digital image;
using the computing device, based on the plurality of key-point sets, determining a plurality of trade-specific activities that appear to be performed by the particular worker;
using the computing device, based on the plurality of trade-specific activities, determining a plurality of trade probabilities, each trade probability among the trade probabilities indicating a likelihood that the particular worker belongs to a particular trade from among a plurality of trades.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
extracting a plurality of background datasets from the plurality of digital images;
based on the plurality of background datasets, determining a plurality of context probabilities, wherein a context probability, of the plurality of context probabilities, indicates a likelihood that a background dataset, of the plurality of background datasets, indicates a trade-specific context of a plurality of trade-specific contexts.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
based on the plurality of trade probabilities and the plurality of context probabilities, generating a plurality of refined probabilities, wherein a refined probability, of the plurality of refined probabilities, indicates a modified likelihood that the particular worker belongs to a trade of the plurality of trades;
selecting, from the plurality of refined probabilities, a highest probability;
based on the highest probability, determining a particular trade to which the particular worker belongs.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
receiving a plurality of unprocessed images;
selecting, from the plurality of unprocessed images, the plurality of digital images that depict the particular worker;
wherein selecting, from the plurality of unprocessed images, the plurality of digital images that depict the particular worker comprises:
detecting a plurality of objects in the plurality of unprocessed images;
wherein detecting a plurality of objects in the plurality of unprocessed images includes one or more of: detecting colors of hard hats depicted in the plurality of unprocessed images, detecting stickers attached to hard hats depicted in the plurality of unprocessed images, recognizing, based on depiction of sensor meshes, positions of workers depicted in the plurality of unprocessed images, or detecting colors of outfits wore by workers depicted in the plurality of unprocessed images;
including, in the plurality of digital images that depict the particular worker, those images in which detected objects include the particular worker;
wherein selecting, from the plurality of unprocessed images, the plurality of digital images that depict the particular worker is performed by executing a convolutional neural network.

### Claim 5 (depends on 4)

The method of claim 4, wherein the determining a plurality of trade probabilities is performed by executing a recurrent neural network;
wherein the plurality of trade probabilities is further determined based on determining, based on the plurality of key-point sets, additional activities depicted in the plurality of digital images;
wherein the additional activities include one or more of: walking, standing, climbing stairs, or sitting.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
determining a plurality of resulting trades for all workers of a plurality of workers;
based on the plurality of resulting trades, determining counts of workers for each trade of the plurality of trades.

### Claim 7 (depends on 2)

The method of claim 2, wherein the plurality of context probabilities is determined by executing a convolutional neural network to perform a patch-based scene segmentation.

### Claim 8

A system comprising one or more processors, one or more non-transitory digital storage media coupled to the processors and storing one or more sequences of program instructions which, when executed using the one or more processors, cause the one or more processors to perform:
using a computing device, receiving a plurality of digital images depicting a particular worker;
using the computing device, based on the plurality of digital images, determining a plurality of key-point sets of the digital images, each of the plurality of key-point sets comprising location information of key points identified within a depiction of the particular worker in a particular digital image;
using the computing device, based on the plurality of key-point sets, determining a plurality of trade-specific activities that appear to be performed by the particular worker;
using the computing device, based on the plurality of trade-specific activities, determining a plurality of trade probabilities, each trade probability among the trade probabilities indicating a likelihood that the particular worker belongs to a particular trade from among a plurality of trades.

### Claim 9 (depends on 8)

The system of claim 8, storing additional instructions for:
extracting a plurality of background datasets from the plurality of digital images;
based on the plurality of background datasets, determining a plurality of context probabilities, wherein a context probability, of the plurality of context probabilities, indicates a likelihood that a background dataset, of the plurality of background datasets, indicates a trade-specific context of a plurality of trade-specific contexts.

### Claim 10 (depends on 9)

The system of claim 9, storing additional program instructions for:
based on the plurality of trade probabilities and the plurality of context probabilities, generating a plurality of refined probabilities, wherein a refined probability, of the plurality of refined probabilities, indicates a modified likelihood that the particular worker belongs to a trade of the plurality of trades;
selecting, from the plurality of refined probabilities, a highest probability;
based on the highest probability, determining a particular trade to which the particular worker belongs.

### Claim 11 (depends on 10)

The system of claim 10, storing additional program instructions for:
receiving a plurality of unprocessed images;
selecting, from the plurality of unprocessed images, the plurality of digital images that depict the particular worker;
wherein selecting, from the plurality of unprocessed images, the plurality of digital images that depict the particular worker comprises:
detecting a plurality of objects in the plurality of unprocessed images;
wherein detecting a plurality of objects in the plurality of unprocessed images includes one or more of: detecting colors of hard hats depicted in the plurality of unprocessed images, detecting stickers attached to hard hats depicted in the plurality of unprocessed images, recognizing, based on depiction of sensor meshes, positions of workers depicted in the plurality of unprocessed images, or detecting colors of outfits wore by workers depicted in the plurality of unprocessed images;
including, in the plurality of digital images that depict the particular worker, those images in which detected objects include the particular worker;
wherein selecting, from the plurality of unprocessed images, the plurality of digital images that depict the particular worker is performed by executing a convolutional neural network.

### Claim 12 (depends on 11)

The system of claim 11, wherein the determining a plurality of trade probabilities is performed by executing a recurrent neural network;
wherein the plurality of trade probabilities is further determined based on determining, based on the plurality of key-point sets, additional activities depicted in the plurality of digital images;
wherein the additional activities include one or more of: walking, standing, climbing stairs, or sitting.

### Claim 13 (depends on 12)

The system of claim 12, storing additional program instructions for:
determining a plurality of resulting trades for all workers of a plurality of workers;
based on the plurality of resulting trades, determining counts of workers for each trade of the plurality of trades.

### Claim 14 (depends on 9)

The system of claim 9, wherein the plurality of context probabilities is determined by executing a convolutional neural network to perform a patch-based scene segmentation.

### Claim 15

One or more non-transitory digital storage media storing one or more sequences of program instructions which, when executed using one or more processors, cause the one or more processors to perform:
using a computing device, receiving a plurality of digital images depicting a particular worker;
using the computing device, based on the plurality of digital images, determining a plurality of key-point sets of the digital images, each of the plurality of key-point sets comprising location information of key points identified within a depiction of the particular worker in a particular digital image;
using the computing device, based on the plurality of key-point sets, determining a plurality of trade-specific activities that appear to be performed by the particular worker;
using the computing device, based on the plurality of trade-specific activities, determining a plurality of trade probabilities, each trade probability among the trade probabilities indicating a likelihood that the particular worker belongs to a particular trade from among a plurality of trades.

### Claim 16 (depends on 15)

The one or more non-transitory digital storage media of claim 15, storing additional instructions for:
extracting a plurality of background datasets from the plurality of digital images;
based on the plurality of background datasets, determining a plurality of context probabilities, wherein a context probability, of the plurality of context probabilities, indicates a likelihood that a background dataset, of the plurality of background datasets, indicates a trade-specific context of a plurality of trade-specific contexts.

### Claim 17 (depends on 16)

The one or more non-transitory digital storage media of claim 16, storing additional program instructions for:
based on the plurality of trade probabilities and the plurality of context probabilities, generating a plurality of refined probabilities, wherein a refined probability, of the plurality of refined probabilities, indicates a modified likelihood that the particular worker belongs to a trade of the plurality of trades;
selecting, from the plurality of refined probabilities, a highest probability;
based on the highest probability, determining a particular trade to which the particular worker belongs.

### Claim 18 (depends on 17)

The one or more non-transitory digital storage media of claim 17, storing additional program instructions for:
receiving a plurality of unprocessed images;
selecting, from the plurality of unprocessed images, the plurality of digital images that depict the particular worker;
wherein selecting, from the plurality of unprocessed images, the plurality of digital images that depict the particular worker comprises:
detecting a plurality of objects in the plurality of unprocessed images;
wherein detecting a plurality of objects in the plurality of unprocessed images includes one or more of: detecting colors of hard hats depicted in the plurality of unprocessed images, detecting stickers attached to hard hats depicted in the plurality of unprocessed images, recognizing, based on depiction of sensor meshes, positions of workers depicted in the plurality of unprocessed images, or detecting colors of outfits wore by workers depicted in the plurality of unprocessed images;
including, in the plurality of digital images that depict the particular worker, those images in which detected objects include the particular worker;
wherein selecting, from the plurality of unprocessed images, the plurality of digital images that depict the particular worker is performed by executing a convolutional neural network.

### Claim 19 (depends on 18)

The one or more non-transitory digital storage media of claim 18, wherein the determining a plurality of trade probabilities is performed by executing a recurrent neural network;
wherein the plurality of trade probabilities is further determined based on determining, based on the plurality of key-point sets, additional activities depicted in the plurality of digital images;
wherein the additional activities include one or more of: walking, standing, climbing stairs, or sitting.

### Claim 20 (depends on 19)

The one or more non-transitory digital storage media of claim 19, storing additional program instructions for:
determining a plurality of resulting trades for all workers of a plurality of workers;
based on the plurality of resulting trades, determining counts of workers for each trade of the plurality of trades.