Systems and methods for in-cabin monitoring with liveliness detection

A system for monitoring an object in a compartment of a vehicle, the system comprising:

• a plurality of time-of-flight sensors in communication with each other and configured to generate a point cloud representing the compartment of the vehicle, the point cloud including three-dimensional positional information of the compartment;
• processing circuitry in communication with the plurality of time-of-flight sensors;
• an object classification unit including one or more neural networks that are in communication with a body pose database, a skeleton model database, and a computer associated with the processing circuitry, wherein the skeleton model database comprises a plurality of keypoints corresponding to various joints of a skeleton, and wherein the body pose database and the skeleton model database cooperate to provide one or more target point clouds comprising target keypoint information that corresponds to target body pose data, and wherein the one or more neural networks are configured to be trained and applied using point cloud data that includes representations of occupants in a plurality of different body poses;
• and
• the processing circuitry configured to:
• compare point clouds generated by each of the plurality of time-of-flight sensors to one another to increase accuracy of the point cloud by having the object classification unit compare the compared point cloud to the one or more target point clouds;
• determine a shape of the object based on the three-dimensional positional information;
• classify the object as an occupant based on the shape;
• estimate a pose of the occupant via comparison of the compared point cloud to the one or more target point clouds;
• identify a body segment of the occupant;
• compare the body segment to target keypoints corresponding to a target attribute for the body segment;
• determine a condition of the occupant based on the comparison of the body segment to the target keypoints;
• and
• generating an output based on the determined condition.

A system for monitoring an object in a compartment of a vehicle, the system comprising:

• a plurality of LiDAR modules in communication with each other and configured to generate a point cloud representing the compartment of the vehicle, the point cloud including three-dimensional positional information of the compartment;
• a powertrain that includes a plurality of powertrain systems that are configured to drive a motion of a vehicle;
• and
• processing circuitry in communication with the plurality of LiDAR modules, the processing circuitry configured to:
• compare point clouds generated by each of the plurality of LiDAR modules to one another to increase accuracy of the point cloud;
• determine a shape of the object based on the three-dimensional positional information;
• classify the object as an occupant based on the shape;
• identify a body segment of the occupant based on the point cloud;
• compare the body segment to target keypoints corresponding to a target attribute for the body segment;
• determine a condition of the occupant based on the comparison of the body segment to the target keypoints including determining a pose of the occupant;
• determine an unfocused state of the occupant based on a comparison of the pose of the occupant to target keypoints of body pose data;
• process the point cloud captured by the plurality of LiDAR modules to determine target operations including target steering angles, rates of motion, and/or speed changes;
• communicate the target operations to the powertrain to allow for at least partially autonomous control over the motion of the vehicle;
• and
• generate an output based on the determined condition, wherein, based on identification of the unfocused state of the occupant, the output comprises communicating signals to adjust the powertrain to stop the vehicle or move the vehicle via the at least partially autonomous control.