# US 12,223,759 B2 · Neural network-based recognition of trade workers present on industrial sites > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,759 B2 - Document ID: US12223759B2 - Status: Granted - Kind: B2 - Application number: 18/400,592 - Filed: 2023-12-29 - Published: 2024-09-12 - Granted: 2025-02-11 - Earliest claimed: 2018-09-19 - Assignees: Procore Technologies, Inc. - Applicants: Procore Technologies, Inc. - Inventors: Lai Him Matthew Man; Mohammad Soltani; Ahmed Aly; Walid Aly - Web page: https://patentgrep.com/patent/US12223759B2 ## CPC - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06V 10/245: {by locating a pattern; Special marks for positioning} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An example computing platform comprising is configured to (i) receive, via one or more cameras positioned on a construction site, a plurality of images, (ii) detect, within the plurality of images, a plurality of objects being worn by respective workers on the construction site, (iii) select, from the plurality of images, a set of images depicting a particular worker, and (iv) based on the selected set of images depicting the particular worker, determine a plurality of trade probabilities for the particular worker, each trade probability in the plurality of trade probabilities indicating a likelihood that the particular worker belongs to a particular trade from among a plurality of trades. ## Claims ### Claim 1 A computing platform comprising: a network interface; at least one processor; non-transitory computer-readable medium; and program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing platform is configured to: receive, via one or more cameras positioned on a construction site, a set of images depicting a particular worker; based on the set of images, determine a plurality of trade probabilities for the particular worker, each trade probability in the plurality of trade probabilities indicating a likelihood that the particular worker belongs to a particular trade from among a plurality of trades; extract a set of background datasets from the set of images; determine, based on the set of background datasets, a set of contextual probabilities, wherein each respective contextual probability of the set of contextual probabilities indicates a likelihood that a respective background dataset of the set of background datasets indicates a trade-specific context from among a plurality of trade-specific contexts; based on the plurality of trade probabilities and the set of contextual probabilities, select a particular trading having a highest probability; and determine, based on the highest probability, that the particular worker belongs to the particular trade. ### Claim 2 (depends on 1) The computing platform of claim 1, wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to receive the set of images depicting the particular worker comprise program instructions that are executable by the at least one processor such that the computing platform is configured to receive, via execution of a convolutional neural network, the set of images depicting the particular worker. ### Claim 3 (depends on 1) The computing platform of claim 1, wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to based on the plurality of trade probabilities and the set of contextual probabilities, select a particular trade having a highest probability comprise program instructions that are executable by the at least one processor such that the computing platform is configured to: based on the plurality of trade probabilities and the set of contextual probabilities, generate a plurality of refined probabilities, wherein a refined probability, of the plurality of refined probabilities, indicates a modified likelihood that the particular worker belongs to the particular trade of the plurality of trades; and select, from the plurality of refined probabilities, a highest probability. ### Claim 4 (depends on 1) The computing platform of claim 1, wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to determine a plurality of trade probabilities for the particular worker comprise program instructions that are executable by the at least one processor such that the computing platform is configured to: detect, within the set of images, one or more objects being worn by the particular worker. ### Claim 5 (depends on 4) The computing platform of claim 4, wherein the one or more objects being worn by respective workers on the construction site comprise one or more of (i) a hard hat of a given color, (ii) a vest of a given color, or (iii) a sticker attached to a hard hat or a vest. ### Claim 6 (depends on 4) The computing platform of claim 4, wherein the set of images depicting the particular worker comprises (i) a first object of the one or more objects being worn by the particular worker, and (ii) a second object of the one or more objects being worn by the particular worker. ### Claim 7 (depends on 6) The computing platform of claim 6, wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to, based on the set of images depicting the particular worker, determine the plurality of trade probabilities for the particular worker comprise program instructions that are executable by the at least one processor such that the computing platform is configured to determine the plurality of trade probabilities for the particular worker based on the first object and the second object, wherein, when worn together, the first object and the second object are indicative of a particular trade of the plurality of trades. ### Claim 8 (depends on 1) The computing platform of claim 1, further comprising program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing platform is configured to: based on the set of images, determine a plurality of key-point sets, each of the plurality of key-point sets comprising location information indicating key points identified within a depiction of the particular worker in a particular image; and based on the plurality of key-point sets, determine a plurality of trade-specific activities that appear to be performed by the particular worker. ### Claim 9 (depends on 1) The computing platform of claim 1, wherein the program instructions that are executable by the at least one processor such that the computing platform is configured to receive the set of images comprise program instructions that are executable by the at least one processor such that the computing platform is configured to receive at least one unprocessed video stream. ### Claim 10 A non-transitory computer-readable medium, wherein the non-transitory computer-readable medium is provisioned with program instructions that, when executed by at least one processor, cause a computing platform to: receive, via one or more cameras positioned on a construction site, a set of images depicting a particular worker; based on the set of images, determine a plurality of trade probabilities for the particular worker, each trade probability in the plurality of trade probabilities indicating a likelihood that the particular worker belongs to a particular trade from among a plurality of trades; extract a set of background datasets from the set of images; determine, based on the set of background datasets, a set of contextual probabilities, wherein each respective contextual probability of the set of contextual probabilities indicates a likelihood that a respective background dataset of the set of background datasets indicates a trade-specific context from among a plurality of trade-specific contexts; based on the plurality of trade probabilities and the set of contextual probabilities, select a particular trading having a highest probability; and determine, based on the highest probability, that the particular worker belongs to the particular trade. ### Claim 11 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the program instructions that, when executed by at least one processor, cause the computing platform to receive the set of images depicting the particular worker comprise program instructions that, when executed by at least one processor, cause the computing platform to receive, via execution of a convolutional neural network, the set of images depicting the particular worker. ### Claim 12 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the program instructions that, when executed by at least one processor, cause the computing platform to, based on the plurality of trade probabilities and the set of contextual probabilities, select a particular trade having a highest probability comprise program instructions that, when executed by at least one processor, cause the computing platform to: based on the plurality of trade probabilities and the set of contextual probabilities, generate a plurality of refined probabilities, wherein a refined probability, of the plurality of refined probabilities, indicates a modified likelihood that the particular worker belongs to the particular trade of the plurality of trades; and select, from the plurality of refined probabilities, a highest probability. ### Claim 13 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the program instructions that, when executed by at least one processor, cause the computing platform to determine a plurality of trade probabilities for the particular worker comprise program instructions that, when executed by at least one processor, cause the computing platform to: detect, within the set of images, one or more objects being worn by the particular worker. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the one or more objects being worn by respective workers on the construction site comprise one or more of (i) a hard hat of a given color, (ii) a vest of a given color, or (iii) a sticker attached to a hard hat or a vest. ### Claim 15 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the set of images depicting the particular worker comprises (i) a first object of the one or more objects being worn by the particular worker, and (ii) a second object of the one or more objects being worn by the particular worker. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the program instructions that, when executed by at least one processor, cause the computing platform to, based on the set of images depicting the particular worker, determine the plurality of trade probabilities for the particular worker comprise program instructions that, when executed by at least one processor, cause the computing platform to determine the plurality of trade probabilities for the particular worker based on the first object and the second object, wherein, when worn together, the first object and the second object are indicative of a particular trade of the plurality of trades. ### Claim 17 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the non-transitory computer-readable medium is also provisioned with program instructions that, when executed by at least one processor, cause the computing platform to: based on the set of images, determine a plurality of key-point sets, each of the plurality of key-point sets comprising location information indicating key points identified within a depiction of the particular worker in a particular image; and based on the plurality of key-point sets, determine a plurality of trade-specific activities that appear to be performed by the particular worker. ### Claim 18 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the program instructions that, when executed by at least one processor, cause the computing platform to receive the set of images comprise program instructions that, when executed by at least one processor, cause the computing platform to receive at least one unprocessed video stream. ### Claim 19 A method carried out by a computing platform, the method comprising: receiving, via one or more cameras positioned on a construction site, a set of images depicting a particular worker; based on the set of images, determining a plurality of trade probabilities for the particular worker, each trade probability in the plurality of trade probabilities indicating a likelihood that the particular worker belongs to a particular trade from among a plurality of trades; extracting a set of background datasets from the set of images; determining, based on the set of background datasets, a set of contextual probabilities, wherein each respective contextual probability of the set of contextual probabilities indicates a likelihood that a respective background dataset of the set of background datasets indicates a trade-specific context from among a plurality of trade-specific contexts; based on the plurality of trade probabilities and the set of contextual probabilities, selecting a particular trading having a highest probability; and determining, based on the highest probability, that the particular worker belongs to the particular trade. ### Claim 20 (depends on 19) The method of claim 19, wherein determining a plurality of trade probabilities for the particular worker comprises: detecting, within the set of images, one or more objects being worn by the particular worker.