# US 11,443,139 B2 · Contextual safety assessment, recommendations, provisioning and monitoring > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,139 B2 - Document ID: US11443139B2 - Status: Granted - Kind: B2 - Application number: 16/887,692 - Filed: 2020-05-29 - Published: 2021-12-02 - Granted: 2022-09-13 - Earliest claimed: 2020-05-29 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Maletsabisa Molapo; Chane Simone Moodley; Sibusisiwe Audrey Makhanya; Darlington Shingirirai Mapiye - Web page: https://patentgrep.com/patent/US11443139B2 ## CPC - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06F 16/903: Querying (for retrieval from the web G06F16/953) - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate - G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30) - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Method and systems for identifying dangers in an accessible environment. Computer vision and video analysis is performed on one or more images and/or video of the accessible environment to generate a series of still frames for a database of training images and natural language processing (NLP) is performed on one or more written reports to generate textual parameters. An online image similarity search is performed using the textual parameters to generate additional images for the database of training images and a risk/danger prediction model is generated based on the database of training images. One or more potential dangers are identified, and one or more warnings are generated using the risk/danger prediction model. ## Claims ### Claim 1 A method to identify dangers in an accessible environment, the method comprising: performing computer vision and video analysis on one or more images and/or video of the accessible environment obtained from one or more cameras to generate a series of still frames for a database of training images; identifying objects and structures in the series of still frames; performing natural language processing (NLP) on one or more written reports to generate textual parameters and match the textual parameters to ones of the objects and the structures and caption respective ones of the still frames including the ones of the objects and the structures; performing an online image similarity search using the textual parameters to generate additional images for the database of training images; generating a risk/danger prediction model based on the database of training images; identifying one or more potential dangers and generating one or more warnings using the risk/danger prediction model; performing NLP and computer vision analysis on the one or more warnings to train a safety proof recommendation model; generating one or more safety upgrade recommendations and one or more cost estimates for one or more safety upgrades based on the safety proof recommendation model; assessing an adequacy of the structures and results of the video analysis to generate a list of high-danger zones in the accessible environment; and triggering an alarm is in response to an individual being observed in a real-time video analysis approaching a high-danger zone in the list of high-danger zones. ### Claim 2 (depends on 1) The method of claim 1, wherein at least one of the one or more warnings has an associated probability of injury. ### Claim 3 (depends on 1) The method of claim 1, further comprising capturing the one or more images and/or video based on a view of the accessible environment from a specified height from a ground level. ### Claim 4 (depends on 3) The method of claim 3, further comprising positioning a camera at the specified height using a depth sensor of the camera that measures a current height of the camera from the ground level. ### Claim 5 (depends on 1) The method of claim 1, further comprising capturing an inventory of the objects in the accessible environment by guiding a user through taking pictures of the accessible environment. ### Claim 6 (depends on 1) The method of claim 1, further comprising generating one or more additional training images using a generative adversarial network (GAN) or an autoencoder. ### Claim 7 (depends on 1) The method of claim 1, further comprising performing computer vision and video analysis on home accident video to generate a second series of still frames for the database of training images; and performing natural language processing and computer vision analysis on the second series of still frames to identify one or more additional objects within the training images and to generate at least one caption for the training images including the one or more additional objects. ### Claim 8 (depends on 7) The method of claim 7, further comprising performing natural language processing to extract captions, objects, and relative parameter associations to generate training text, and wherein the generation of the risk/danger prediction model is based on the training text. ### Claim 9 (depends on 1) The method of claim 1, further comprising: generating an individual-specific risk assessment model based on the one or more training images and the textual parameters; and identifying one or more individual specific risks using the individual-specific risk assessment model. ### Claim 10 (depends on 1) The method of claim 1, wherein the training of the safety proof recommendation model uses individual specific risks and live cost updates. ### Claim 11 (depends on 1) The method of claim 1, further comprising issuing a notification and an alert to a user regarding the high-danger zone. ### Claim 12 (depends on 1) The method of claim 1, further comprising: tracking movement of an individual via one or more cameras; performing computer vision and video analysis to generate one or more still frames and object identification; performing behavioral analysis on the still frames and the object identification to identify one or more personality traits of the individual; and identifying the one or more potential dangers based on the one or more personality traits. ### Claim 13 (depends on 12) The method of claim 12, further comprising detecting an anomaly which could trigger an adverse response in the individual. ### Claim 14 A non-transitory computer readable medium comprising computer executable instructions which when executed by a computer cause the computer to perform a method of: performing computer vision and video analysis on one or more images and/or video of an accessible environment obtained from one or more cameras to generate a series of still frames for a database of training images; identifying objects and structures in the series of still frames; performing natural language processing (NLP) on one or more written reports to generate textual parameters and match the textual parameters to ones of the objects and the structures and caption respective ones of the still frames including the ones of the objects and the structures; performing an online image similarity search using the textual parameters to generate additional images for the database of training images; generating a risk/danger prediction model based on the database of training images; identifying one or more potential dangers and generating one or more warnings using the risk/danger prediction model; performing NLP and computer vision analysis on the one or more warnings to train a safety proof recommendation model; generating one or more safety upgrade recommendations and one or more cost estimates for one or more safety upgrades based on the safety proof recommendation model; assessing an adequacy of the structures and results of the video analysis to generate a list of high-danger zones in the accessible environment; and triggering an alarm is in response to an individual being observed in a real-time video analysis approaching a high-danger zone in the list of high-danger zones. ### Claim 15 An apparatus comprising: a memory; and at least one processor, coupled to said memory, and operative to perform operations comprising: performing computer vision and video analysis on one or more images and/or video of an accessible environment obtained from one or more cameras to generate a series of still frames for a database of training images; identifying objects and structures in the series of still frames; performing natural language processing (NLP) on one or more written reports to generate textual parameters and match the textual parameters to ones of the objects and the structures and caption respective ones of the still frames including the ones of the objects and the structures; performing an online image similarity search using the textual parameters to generate additional images for the database of training images; generating a risk/danger prediction model based on the database of training images; identifying one or more potential dangers and generating one or more warnings using the risk/danger prediction model; performing NLP and computer vision analysis on the one or more warnings to train a safety proof recommendation model; generating one or more safety upgrade recommendations and one or more cost estimates for one or more safety upgrades based on the safety proof recommendation model; assessing an adequacy of the structures and results of the video analysis to generate a list of high-danger zones in the accessible environment; and triggering an alarm is in response to an individual being observed in a real-time video analysis approaching a high-danger zone in the list of high-danger zones. ### Claim 16 (depends on 15) The apparatus of claim 15, the operations further comprising capturing the one or more images and/or video based on a view of the accessible environment from a perspective of a specified individual. ### Claim 17 (depends on 15) The apparatus of claim 15, the operations further comprising recommending a height to position a camera and using a depth sensor to measure a height of the camera from ground level. ### Claim 18 (depends on 15) The apparatus of claim 15, the operations further comprising performing computer vision and video analysis on home accident video to generate a second series of still frames for the database of training images; and performing natural language processing and computer vision analysis on the second series of still frames to identify one or more additional objects within the training images and to generate at least one caption for the training images including the one or more additional objects.