# US 2021/0276185 A1 · Imaging process for detecting failures modes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0276185 A1
- Document ID: US20210276185A1
- Status: Published application
- Kind: A1
- Application number: 17/193,875
- Filed: 2021-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: Embodied Intelligence Inc.
- Applicants: Embodied Intelligence Inc.
- Inventors: Yide Shentu; David Mascharka; Tianhao Zhang; Yan Duan; Jasmine Deng; Xi Chen
- Web page: https://patentgrep.com/patent/US20210276185A1

## CPC

- B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis}
- B25J 9/1669: {characterised by special application, e.g. multi-arm co-operation, assembly, grasping}
- B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement}
- B25J 9/1697: {Vision controlled systems}
- G06N 3/02: Neural networks
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Various embodiments of the present technology generally relate to robotic devices, artificial intelligence, and computer vision. More specifically, some embodiments relate to an imaging process for detecting failure modes in a robotic motion environment. In one embodiment, a method of detecting failure modes in a robotic motion environment comprises collecting one or more images of a multiple scenes throughout a robotic motion cycle. Images may be collected by one or more cameras positioned at one or more locations for collecting images with various views. Images collected throughout the robotic motion cycle may be processed in real-time to determine if any failure modes are present in their respective scenes, report when failure modes are present, and may be used to direct a robotic device accordingly.

## Claims

### Claim 1

A method of detecting failure modes in a robotic motion environment, the method comprising:
collecting one or more first images of a first scene while a robot is in a first position; and
determining if a failure mode of a first set of failure modes is present in the first scene based on the one or more first images;
when the failure mode of the first set of failure modes is not present in the first scene:
collecting one or more intermediary images of an intermediary scene while the robot is in an intermediary position; and
determining if a failure mode of a second set of failure modes is present in the intermediary scene based on the one or more intermediary images;
when the failure mode of the second set of failure modes is not present in the intermediary scene:
collecting one or more final images of a final scene while the robot is in a final position; and
determining if a failure mode of a third set of failure modes is present in the final scene based on the one or more final images.

### Claim 2 (depends on 1)

The method of claim 1, further comprising
producing a segmentation prediction based on the one or more first images, wherein the segmentation prediction identifies one or more distinct objects in the first scene; and
determining if the failure mode of the first set of failure modes is present in the first scene based on the segmentation prediction.

### Claim 3 (depends on 1)

The method of claim 1, wherein the intermediary scene comprises at least a portion of the robot.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first set of failure modes comprises at least one of: a failure more in which no objects are present, a failure mode in which no objects are suitable for interaction with the robot, and a failure mode in which the robot is in a position that will cause an error.

### Claim 5 (depends on 1)

The method of claim 1, wherein the robot is a robotic arm for picking and placing objects.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
the first scene comprises one or more objects for the robot to pick up;
the one or more first images of the first scene are collected before the robot attempts to pick up an object of the one or more objects from the first scene;
the one or more intermediary images of the intermediary scene are collected after the robot attempts to pick up the object of the one or more objects from the first scene;
the intermediary scene comprises at least a portion of the robot and at least one object of the one or more objects from the first scene picked up by the robot;
the one or more final images of the final scene are collected after the robot attempts to place the one or more objects from the first scene picked up by the robot; and
the final scene comprises an area in which the robot attempts to place the one or more objects from the first scene picked up by the robot.

### Claim 7 (depends on 6)

The method of claim 6, wherein the second set of failure modes comprises at least one of: a failure mode in which the robot picked up two objects, a failure mode in which an object picked up by the robot is damaged, a failure mode in which an object is deformed after being picked up by the robot, and a failure mode in which the robot is in a position that will cause an error.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
when the failure mode of the first set of failure modes is present in the first scene, reporting that the failure mode of the first set of failure modes is present;
when the failure mode of the second set of failure modes is present in the intermediary scene, reporting that the failure mode of the second set of failure modes is present; and
when the failure mode of the third set of failure modes is presented in the final scene, reporting that the failure mode of the third set of failure modes is present.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
when the failure mode of the first set of failure modes is present in the first scene, resolving an error associated with the failure mode of the first set of failure modes;
when the failure mode of the second set of failure modes is present in the intermediary scene, resolving an error associated with the failure mode of the second set of failure modes; and
when the failure mode of the third set of failure modes is presented in the final scene, resolving an error associated with the failure mode of the third set of failure modes.

### Claim 10

A failure mode detection system comprising:
one or more computer-readable storage media;
a processing system operatively coupled to the one or more computer-readable storage media; and
program instructions, stored on the one or more computer-readable storage media, wherein the program instructions, when read and executed by the processing system, direct the processing system to:
collect one or more first images of a first scene while a robot is in a first position; and
determine if a failure mode of a first set of failure modes is present in the first scene based on the one or more first images;
when the failure mode of the first set of failure modes is not present in the first scene:
collect one or more intermediary images of an intermediary scene while the robot is in an intermediary position; and
determine if a failure mode of a second set of failure modes is present in the intermediary scene based on the one or more intermediary images;
when the failure mode of the second set of failure modes is not present in the intermediary scene:
collect one or more final images of a final scene while the robot is in a final position; and
determine if a failure mode of a third set of failure modes is present in the final based on the one or more final images.

### Claim 11 (depends on 10)

The failure mode detection system of claim 10, wherein the program instructions, when read and executed by the processing system, further direct the processing system to:
produce a segmentation prediction based on the one or more first images, wherein the segmentation prediction identifies one or more distinct objects in the first scene; and
determine if the failure mode of the first set of failure modes is present in the first scene based on the segmentation prediction.

### Claim 12 (depends on 10)

The failure mode detection system of claim 10, wherein the first set of failure modes comprises at least one of: a failure more in which no objects are present, a failure mode in which no objects are suitable for interaction with the robot, and a failure mode in which the robot is in a position that will cause an error.

### Claim 13 (depends on 10)

The failure mode detection system of claim 10, wherein the robot is a robotic arm for picking and placing objects.

### Claim 14 (depends on 13)

The failure mode detection system of claim 13, wherein:
the first scene comprises one or more objects for the robot to pick up;
the one or more first images of the first scene are collected before the robot attempts to pick up an object of the one or more objects from the first scene;
the one or more intermediary images of the intermediary scene are collected after the robot attempts to pick up the object of the one or more objects from the first scene;
the intermediary scene comprises at least a portion of the robot and at least one object of the one or more objects from the first scene picked up by the robot;
the one or more final images of the final scene are collected after the robot attempts to place the one or more objects from the first scene picked up by the robot; and
the final scene comprises an area in which the robot attempts to place the one or more objects from the first scene picked up by the robot.

### Claim 15 (depends on 14)

The failure mode detection system of claim 14, wherein the second set of failure modes comprises at least one of: a failure mode in which the robot picked up two objects, a failure mode in which an object picked up by the robot is damaged, a failure mode in which an object is deformed after being picked up by the robot, and a failure mode in which the robot is in a position that will cause an error.

### Claim 16 (depends on 10)

The failure mode detection system of claim 10, wherein the program instructions, when read and executed by the processing system, further direct the processing system to:
when the failure mode of the first set of failure modes is present in the first scene, report that the failure mode of the first set of failure modes is present;
when the failure mode of the second set of failure modes is present in the intermediary scene, report that the failure mode of the second set of failure modes is present; and
when the failure mode of the third set of failure modes is presented in the final scene, report that the failure mode of the third set of failure modes is present.

### Claim 17 (depends on 10)

The failure mode detection system of claim 10, wherein the program instructions, when read and executed by the processing system, further direct the processing system to:
when the failure mode of the first set of failure modes is present in the first scene, resolve an error associated with the failure mode of the first set of failure modes;
when the failure mode of the second set of failure modes is present in the intermediary scene, resolve an error associated with the failure mode of the second set of failure modes; and
when the failure mode of the third set of failure modes is presented in the final scene, resolve an error associated with the failure mode of the third set of failure modes.

### Claim 18

A failure mode detection system, the system comprising:
at least one robot configured to perform a task;
a computer-vision system configured to capture one or more images at multiple stages of the task;
program instructions, stored on one or more computer-readable storage media, wherein the program instructions, when read and executed by a processing system, direct the processing system to:
collect one or more first images of a first scene while a robot is in a first position; and
determine if a failure mode of a first set of failure modes is present in the first scene based on the one or more first images;
when the failure mode of the first set of failure modes is not present in the first scene:
collect one or more intermediary images of an intermediary scene while the robot is in an intermediary position; and
determine if a failure mode of a second set of failure modes is present in the intermediary scene based on the one or more intermediary images;
when the failure mode of the second set of failure modes is not present in the intermediary scene:
collect one or more final images of a final scene while the robot is in a final position; and
determine if a failure mode of a third set of failure modes is present in the final based on the one or more final images.

### Claim 19 (depends on 18)

The failure mode detection system of claim 18, wherein the program instructions, when read and executed by the processing system, further direct the processing system to:
produce a segmentation prediction based on the one or more first images, wherein the segmentation prediction identifies one or more distinct objects in the first scene;
and determine if the failure mode of the first set of failure modes is present in the first scene based on the segmentation prediction.

### Claim 20 (depends on 18)

The failure mode detection system of claim 18, wherein the first set of failure modes comprises at least one of: a failure more in which no objects are present, a failure mode in which no objects are suitable for interaction with the robot, and a failure mode in which the robot is in a position that will cause an error.
