# US 2023/0069482 A1 · Machine learning-based environment fail-safes through multiple camera views

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0069482 A1
- Document ID: US20230069482A1
- Status: Published application
- Kind: A1
- Application number: 17/462,752
- Filed: 2021-08-31
- Published: 2023-03-02
- Earliest claimed: 2021-08-31
- Assignees: Siemens Aktiengesellschaft
- Applicants: Siemens Aktiengesellschaft
- Inventors: Mayuri Deshpande; Anant Kumar Mishra
- Web page: https://patentgrep.com/patent/US20230069482A1

## CPC

- B25J 9/1676: {Avoiding collision or forbidden zones}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 7/77: using statistical methods
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30232: Surveillance
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A computing system may include a fail-safe learning engine configured to access camera data captured by multiple cameras positioned within an environment during a learning phase, generate training data based on the camera data captured by the multiple cameras, and construct a human detection model using the training data. The computing system may also include a fail-safe trigger engine configured to access camera data captured by the multiple cameras positioned within the environment during an active phase, and the camera data captured during the active phase may include a target object. The fail-trigger engine may further be configured to provide, as an input to the human detection model, the camera data that includes the target object and execute a fail-safe action in the environment responsive to the determination, provided by the human detection model, indicating that the target object is a human.

## Claims

### Claim 1

A system comprising:
a fail-safe learning engine configured to:
access camera data captured by multiple cameras positioned within an environment during a learning phase;
generate training data based on the camera data captured by the multiple cameras; and
construct a human detection model using the training data, the human detection model configured to provide a determination indicative of whether a given object within a view of a camera in the environment is a human or not; and
a fail-safe trigger engine configured to:
access camera data captured by the multiple cameras positioned within the environment during an active phase, wherein the camera data captured during the active phase includes a target object;
provide, as an input to the human detection model, the camera data that includes the target object; and
execute a fail-safe action in the environment responsive to the determination, provided by the human detection model, indicating that the target object is a human.

### Claim 2 (depends on 1)

The system of claim 1, wherein the fail-safe trigger engine is configured to execute the fail-safe action in the environment responsive to both:
the determination, provided by the human detection model, indicating that the target object is a human; and
a determination that a position of the target object within the environment is located within a threshold distance from a fail-safe boundary in the environment.

### Claim 3 (depends on 2)

The system of claim 2, wherein the fail-safe trigger engine is further configured to determine the position of the target object within the environment through a triangulation process via the camera data captured from the multiple cameras in the environment.

### Claim 4 (depends on 3)

The system of claim 3, wherein the fail-safe trigger engine is configured to perform the triangulation process by:
accessing a first 2-dimensional (2D) camera frame of the camera data that depicts the target object, wherein the first 2D camera frame is captured by a first camera of the multiple cameras;
accessing a second 2D camera frame of the camera data that depicts the target object, wherein the second 2D camera frame is captured by a second camera of the multiple cameras different from the first camera; and
triangulating a 3-dimensional position of the target object from the first 2D camera frame and the second 2D camera frame based on positioning of the first and second cameras within the environment.

### Claim 5 (depends on 1)

The system of claim 1, wherein the determination provided by the human detection model is a probability indicator indicative of a probability that the given object within a view of a camera in the environment is a human or not.

### Claim 6 (depends on 5)

The system of claim 5, wherein the fail-safe trigger engine configured to:
access primary camera data captured by a primary camera of the multiple cameras, the primary camera data including the target object;
provide, as the input to the human detection model, the primary camera data;
in response to the probability indicator, provided by the human detection model for the primary camera data, indicating that the probability that the target object is a human is greater than a probabilistic threshold:
access secondary camera data captured by a secondary camera of the multiple cameras, the secondary camera data including the target object at a different perspective from the primary camera data;
provide, as the input to the human detection model, the secondary camera data; and
execute the fail-safe action in the environment responsive to the probability indicator, provided by the human detection model for the secondary camera data, indicating that the probability that the target object is a human is also greater than the probabilistic threshold.

### Claim 7 (depends on 5)

The system of claim 5, wherein the fail-safe trigger engine configured to:
access primary camera data captured by a primary camera of the multiple cameras, the primary camera data including the target object;
provide, as the input to the human detection model, the primary camera data;
in response to the probability indicator, provided by the human detection model for the primary camera data, indicating that the probability that the target object is a human is greater than a probabilistic threshold:
access secondary camera data captured by a secondary camera of the multiple cameras, the secondary camera data including the target object at a different perspective from the primary camera data;
provide, as the input to the human detection model, the secondary camera data; and
execute the fail-safe action in the environment responsive to the probability indicator, provided by the human detection model for the secondary camera data, indicating that the probability that the target object is a human is greater than a secondary probabilistic threshold different from the probabilistic threshold applied for the primary camera data.

### Claim 8

A method comprising:
by a computing system:
accessing camera data captured by multiple cameras positioned within an environment during an active phase, wherein the camera data captured during the active phase includes a target object;
providing the camera data that includes the target object as an input to a human detection model, wherein the human detection model is configured to provide a determination indicative of whether a given object within a view of a camera in the environment is a human or not; and
executing a fail-safe action in the environment responsive to the determination, provided by the human detection model, indicating that the target object is a human.

### Claim 9 (depends on 8)

The method of claim 8, comprising executing the fail-safe action in the environment responsive to both:
the determination, provided by the human detection model, indicating that the target object is a human; and
a determination that a position of the target object within the environment is located within a threshold distance from a fail-safe boundary in the environment.

### Claim 10 (depends on 9)

The method of claim 9, further comprising determining the position of the target object within the environment through a triangulation process via the camera data captured from the multiple cameras in the environment.

### Claim 11 (depends on 10)

The method of claim 10, comprising performing the triangulation process by:
accessing a first 2-dimensional (2D) camera frame of the camera data that depicts the target object, wherein the first 2D camera frame is captured by a first camera of the multiple cameras;
accessing a second 2D camera frame of the camera data that depicts the target object, wherein the second 2D camera frame is captured by a second camera of the multiple cameras different from the first camera; and
triangulating a 3-dimensional position of the target object from the first 2D camera frame and the second 2D camera frame based on positioning of the first and second cameras within the environment.

### Claim 12 (depends on 8)

The method of claim 8, wherein the determination provided by the human detection model is a probability indicator indicative of a probability that the given object within a view of a camera in the environment is a human or not.

### Claim 13 (depends on 12)

The method of claim 12, comprising:
accessing primary camera data captured by a primary camera of the multiple cameras, the primary camera data including the target object;
providing, as the input to the human detection model, the primary camera data;
in response to the probability indicator, provided by the human detection model for the primary camera data, indicating that the probability that the target object is a human is greater than a probabilistic threshold:
accessing secondary camera data captured by a secondary camera of the multiple cameras, the secondary camera data including the target object at a different perspective from the primary camera data;
providing, as the input to the human detection model, the secondary camera data; and
executing the fail-safe action in the environment responsive to the probability indicator, provided by the human detection model for the secondary camera data, indicating that the probability that the target object is a human is also greater than the probabilistic threshold.

### Claim 14 (depends on 8)

The method of claim 8, further comprising:
accessing camera data captured by the multiple cameras positioned within the environment during a learning phase;
generating training data based on the camera data captured by the multiple cameras during the learning phase; and
constructing the human detection model using the training data.

### Claim 15

A non-transitory machine-readable medium comprising instructions that, when executed by a processor, cause a computing system to:
access camera data captured by multiple cameras positioned within an environment during an active phase, wherein the camera data captured during the active phase includes a target object;
provide, as an input to a human detection model, the camera data that includes the target object, wherein the human detection model is configured to provide a determination indicative of whether a given object within a view of a camera in the environment is a human or not; and
execute a fail-safe action in the environment responsive to the determination, provided by the human detection model, indicating that the target object is a human.

### Claim 16 (depends on 15)

The non-transitory machine-readable medium of claim 15, wherein the instructions cause the computing system to execute the fail-safe action in the environment responsive to both:
the determination, provided by the human detection model, indicating that the target object is a human; and
a determination that a position of the target object within the environment is located within a threshold distance from a fail-safe boundary in the environment.

### Claim 17 (depends on 16)

The non-transitory machine-readable medium of claim 16, wherein the instructions cause the computing system to determine the position of the target object within the environment through a triangulation process via the camera data captured from the multiple cameras in the environment.

### Claim 18 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein the instructions cause the computing system to perform the triangulation process by:
accessing a first 2-dimensional (2D) camera frame of the camera data that depicts the target object, wherein the first 2D camera frame is captured by a first camera of the multiple cameras;
accessing a second 2D camera frame of the camera data that depicts the target object, wherein the second 2D camera frame is captured by a second camera of the multiple cameras different from the first camera; and
triangulating a 3-dimensional position of the target object from the first 2D camera frame and the second 2D camera frame based on positioning of the first and second cameras within the environment.

### Claim 19 (depends on 15)

The non-transitory machine-readable medium of claim 15, wherein the determination provided by the human detection model is a probability indicator indicative of a probability that the given object within a view of a camera in the environment is a human or not.

### Claim 20 (depends on 19)

The non-transitory machine-readable medium of claim 19, wherein the instructions cause the computing system to:
access primary camera data captured by a primary camera of the multiple cameras, the primary camera data including the target object;
provide, as the input to the human detection model, the primary camera data;
in response to the probability indicator, provided by the human detection model for the primary camera data, indicating that the probability that the target object is a human is greater than a probabilistic threshold:
access secondary camera data captured by a secondary camera of the multiple cameras, the secondary camera data including the target object at a different perspective from the primary camera data;
provide, as the input to the human detection model, the secondary camera data; and
execute the fail-safe action in the environment responsive to the probability indicator, provided by the human detection model for the secondary camera data, indicating that the probability that the target object is a human is also greater than the probabilistic threshold.
