# US 12,602,952 B2 · Image-based automated ergonomic risk root cause and solution identification system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,952 B2 - Document ID: US12602952B2 - Status: Granted - Kind: B2 - Application number: 18/742,633 - Filed: 2024-06-13 - Published: 2025-06-12 - Granted: 2026-04-14 - Earliest claimed: 2023-12-11 - Assignees: VelocityEHS Holdings, Inc. - Applicants: VelocityEHS Holdings, Inc. - Inventors: Julia Penfield; Gunwoo Yong; Meiyin Liu; SangHyun Lee - Web page: https://patentgrep.com/patent/US12602952B2 ## CPC - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Disclosed herein is an image-based system configured to identify root causes of industrial ergonomic risks and their corresponding solutions. An example system comprises a computing device configured to encode an image of a worker performing a work task to generate an embedding vector, transmit the embedding vector to an image-grounded text decoder, while generating first tokens to instruct the decoder to generate a first sentence indicating a root cause of an ergonomic risk identified in the image, compute first relative sensitivity scores relating to the first tokens and extracted image features, generate second tokens of the first sentence based on the first relative sensitivity scores, while generating third tokens to instruct a text decoder to generate a second sentence indicating a solution to the ergonomic risk, calculate second relative sensitivity scores relating to the second and third tokens, and generate the first and second sentences accordingly. ## Claims ### Claim 1 A system, comprising: a computing device including: a non-transitory computer-readable storage medium storing machine readable instructions; and a processor coupled to the non-transitory computer-readable storage medium and configured to execute the machine readable instructions to: obtain an image of a worker performing a work task, encode the image to generate an embedding vector, incorporate a deep learning based model to process the embedding vector to generate a first sentence indicating a root cause of an ergonomic risk identified in the image and a second sentence indicating a solution to the ergonomic risk identified in the image, and establish a dataset to train the deep learning based model by: collecting samples of image-caption data, the image-caption data including a plurality of images and information relating to ergonomic problems identified in the plurality of images and corresponding solutions to the ergonomic problems, categorizing the samples of image-caption data into a plurality of ergonomic problem-solution pairs, identifying, from the samples, training images associated with each of the plurality of ergonomic problem-solution pairs, sampling the plurality of ergonomic problem-solution pairs in connection with the training images, establishing the dataset based at least upon sampling results of the plurality of ergonomic problem-solution pairs in connection with the training images, and training the deep learning based model using the dataset. ### Claim 2 (depends on 1) The system of claim 1, wherein the processor is configured to execute the machine readable instructions to sample the plurality of ergonomic problem-solution pairs in connection with the training images by: oversampling a first portion of the plurality of ergonomic problem-solution pairs in response to identifying that first plurality of the training images associated with the first portion are less than a selected value; and undersampling a second portion of the plurality of ergonomic problem-solution pairs in response to identifying that second plurality of training images associated with the second portion are greater than the selected value. ### Claim 3 (depends on 2) The system of claim 2, wherein the processor is configured to execute the machine readable instructions to sample the plurality of ergonomic problem-solution pairs in connection with the training images by: removing background images from each of the training images; and extracting foreground images from each of the training images. ### Claim 4 (depends on 1) The system of claim 1, wherein the processor is further configured to execute the machine readable instructions to generate multiple captions for each of the plurality of ergonomic problem-solution pairs using a large language model. ### Claim 5 (depends on 4) The system of claim 4, wherein each of the captions includes a first part describing an ergonomic problem and a second part describing an ergonomic solution. ### Claim 6 (depends on 5) The system of claim 5, wherein the processor is further configured to execute the machine readable instructions to prompt the large language model to generate the captions in a plurality of different semantic styles with the same meaning. ### Claim 7 (depends on 1) The system of claim 1, wherein the processor is further configured to execute the machine readable instructions to determine a training loss variation of the deep learning based model over a selected number of epochs. ### Claim 8 A computer-implemented method performed by a computing server system, the computer-implemented method comprising: obtaining, by a processor of the computing server system, an image of a worker performing a work task; encoding the image to generate an embedding vector; incorporating a deep learning based model to process the embedding vector to generate a first sentence indicating a root cause of an ergonomic risk identified in the image and a second sentence indicating a solution to the ergonomic risk identified in the image; and establishing a dataset to train the deep learning based model by: collecting samples of image-caption data, the image-caption data including a plurality of images and information relating to ergonomic problems identified in the plurality of images and corresponding solutions to the ergonomic problems, categorizing the samples of image-caption data into a plurality of ergonomic problem-solution pairs, identifying, from the samples, training images associated with each of the plurality of ergonomic problem-solution pairs, sampling the plurality of ergonomic problem-solution pairs in connection with the training images, establishing the dataset based at least upon sampling results of the plurality of ergonomic problem-solution pairs in connection with the training images, and training the deep learning based model using the dataset. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the sampling the plurality of ergonomic problem-solution pairs in connection with the training images comprises: oversampling a first portion of the plurality of ergonomic problem-solution pairs in response to identifying that first plurality of the training images associated with the first portion are less than a selected value; and undersampling a second portion of the plurality of ergonomic problem-solution pairs in response to identifying that second plurality of training images associated with the second portion are greater than the selected value. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein the sampling the plurality of ergonomic problem-solution pairs in connection with the training images comprises: removing background images from each of the training images; and extracting foreground images from each of the training images. ### Claim 11 (depends on 8) The computer-implemented method of claim 8, further comprising generating multiple captions for each of the plurality of ergonomic problem-solution pairs using a large language model. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein each of the captions includes a first part describing an ergonomic problem and a second part describing an ergonomic solution. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein the captions are generated by the large language model in a plurality of different semantic styles with the same meaning. ### Claim 14 (depends on 8) The computer-implemented method of claim 8, further comprising determining a training loss variation of the deep learning based model over a selected number of epochs. ### Claim 15 A non-transitory computer readable medium storing machine executable instructions for a computing server system, the machine executable instructions being configured for: obtaining, by a processor of the computing server system, an image of a worker performing a work task; encoding the image to generate an embedding vector; incorporating a deep learning based model to process the embedding vector to generate a first sentence indicating a root cause of an ergonomic risk identified in the image and a second sentence indicating a solution to the ergonomic risk identified in the image; and establishing a dataset to train the deep learning based model by: collecting samples of image-caption data, the image-caption data including a plurality of images and information relating to ergonomic problems identified in the plurality of images and corresponding solutions to the ergonomic problems, categorizing the samples of image-caption data into a plurality of ergonomic problem-solution pairs, identifying, from the samples, training images associated with each of the plurality of ergonomic problem-solution pairs, sampling the plurality of ergonomic problem-solution pairs in connection with the training images, establishing the dataset based at least upon sampling results of the plurality of ergonomic problem-solution pairs in connection with the training images, and training the deep learning based model using the dataset. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the instructions for the sampling the plurality of ergonomic problem-solution pairs in connection with the training images further comprise instructions for: oversampling a first portion of the plurality of ergonomic problem-solution pairs in response to identifying that first plurality of the training images associated with the first portion are less than a selected value; and undersampling a second portion of the plurality of ergonomic problem-solution pairs in response to identifying that second plurality of training images associated with the second portion are greater than the selected value. ### Claim 17 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the instructions for the sampling the plurality of ergonomic problem-solution pairs in connection with the training images further comprise instructions for: removing background images from each of the training images; and extracting foreground images from each of the training images. ### Claim 18 (depends on 15) The non-transitory computer readable medium of claim 15, further comprising instructions for generating multiple captions for each of the plurality of ergonomic problem-solution pairs using a large language model. ### Claim 19 (depends on 18) The non-transitory computer readable medium of claim 18, wherein each of the captions includes a first part describing an ergonomic problem and a second part describing an ergonomic solution, and the captions are generated by the large language model in a plurality of different semantic styles with the same meaning. ### Claim 20 (depends on 15) The non-transitory computer readable medium of claim 15, further comprising instructions for determining a training loss variation of the deep learning based model over a selected number of epochs.