Camera-based boom control

A system comprising:

• a machine including
• a machine body,
• an articulating boom arm including a hoist boom coupled to the machine body and a stick boom coupled to a distal end of the hoist boom,
• a hoist actuator configured to controllably adjust an angle of the hoist boom relative to the machine body, and
• a stick actuator configured to controllably adjust an angle of the stick boom relative to the hoist boom;
• a hoist sensor configured to output a signal indicative of the angle of the hoist boom relative to the machine body that is directly measured by the hoist sensor;
• a stick sensor configured to output a signal indicative of the angle of the stick boom relative to the hoist boom that is directly measured by the stick sensor;
• a camera mounted to the machine with a field of view that includes at least a part of the machine;
• an electronic processor configured to
• train a neural network to determine, based on image data captured by the camera, at least one selected from a group consisting of the angle of the hoist boom relative to the machine body and the angle of the stick boom relative to the hoist boom, wherein the signal output by the hoist sensor, the signal output by the stick sensor, and an image captured by the camera are used as training input to train the neural network;
• determine an actual pose of the articulating boom arm based on at least one selected from a group consisting of the signal output from the hoist sensor, the signal output by the stick sensor, and the output of the neural network based on image data captured by the camera;
• and
• operate the hoist actuator and the stick actuator based at least in part on the determined actual pose of the articulating boom arm,
• wherein at least a portion of the machine includes a reflective paint, and wherein the electronic processor is further configured to filter the image data from the camera to distinguish the reflective paint on the machine from an image background before providing the captured image data as input to the neural network.

A method for controlling movement of an articulating boom arm of a machine, the method comprising:

• determining whether a position sensor is operating properly, wherein the position sensor is configured to directly measure a pose of at least one part of the articulating boom;
• determining an actual pose of the articulating boom arm based on a signal output of the position sensor in response to determining that the position sensor is operating properly;
• determining the actual pose of the articulating boom arm based on an artificial intelligence mechanism in response to determining that the position sensor is not operating properly, wherein determining the actual pose of the articulating boom arm based on the artificial intelligence mechanism includes:
• capturing at least one image with a camera mounted on the machine, wherein the field of view of the camera includes at least a portion of the machine;
• applying the artificial intelligence mechanism with the at least one image as an input to the artificial intelligence mechanism, wherein the artificial intelligence mechanism is trained to output at least one value indicative of a pose of the articulating boom arm based on the at least one image as the input;
• and
• operating an actuator configured to cause movement of the articulating boom arm based at least in part on the determined actual pose of the articulating boom arm.