Back to results
US 11,123,876 B2Granted
Method for sensor data processing
AssigneeCaterpillar Inc.
InventorsKevin Peterson, Matt Delaney, Zachary Witte, Jason Calaiaro
Life of this patent
Estimating the term…
Abstract
A method for remote operation of a robot, preferably including: recording a set of sensor streams; transmitting a transmission stream; selecting a received stream for display; and/or displaying the selected stream. A system, preferably including a robot and a remote operation system connected to the robot by one or more communication networks.
Claims20
A method for remote operation of a robot, comprising:
- at a first camera of the robot, capturing a first video stream;
- substantially concurrent with capturing the first video stream, at a second camera of the robot, capturing a second video stream;
- at the robot:
- processing the first video stream to generate a first processed video stream;
- processing the second video stream to generate a second processed video stream, comprising performing a bitrate reduction process;
- combining the first and second processed video streams into a first combined video stream;
- and
- transmitting the first combined video stream to a remote operation system, said transmitting including transmitting the first combined video stream and at least one redundant copy of the first combined video stream to the remote operation system;
- at the remote operation system:
- receiving the first combined video stream and the at least one redundant copy of the first combined video stream from the robot;
- in response to receiving the first combined video stream and the at least one redundant copy of the first combined video stream, dynamically selecting one of the first combined video stream or the at least one redundant copy of the first combined video stream and displaying the selected combined video stream to an operator;
- while displaying the selected combined video stream, receiving a control input from the operator;
- in response to receiving the control input, generating a control instruction based on the control input;
- and
- in response to generating the control instruction, transmitting the control instruction to the robot;
- at the robot:
- receiving the control instruction from the remote operation system;
- in response to receiving the control instruction, controlling robot operation based on the control instruction;
- after controlling robot operation based on the control instruction, sampling a measurement;
- based on the measurement, determining an increased priority state associated with the second camera;
- after determining the increased priority state, at the first camera, capturing a third video stream;
- substantially concurrent with capturing the third video stream, at the second camera, capturing a fourth video stream;
- based on the increased priority state, processing the third video stream to generate a third processed video stream, wherein a first processed video stream bitrate is not less than a third processed video stream bitrate;
- based on the increased priority state, processing the fourth video stream to generate a fourth processed video stream, wherein a second processed video stream bitrate is less than a fourth processed video stream bitrate;
- combining the third and fourth processed video streams into a second combined video stream;
- and
- transmitting the second combined video stream to the remote operation system.
24 more elements
A system for remote operation of a robot, comprising:
- control circuitry of the robot configured to:
- process a first video stream captured using a first camera of the robot to generate a first processed video stream,
- process a second video stream, captured substantially concurrent with the first video stream using a second camera of the robot, to generate a second processed video stream, comprising performing a bitrate reduction process,
- combine the first and second processed video streams into a first combined video stream,
- transmit the first combined video stream to a remote operation system,
- responsive to receiving a control instruction from the remote operation system, control robot operation based on the control instruction,
- sample a measurement after controlling the robot operation based on the control instruction,
- determine an increased priority state associated with the second camera based on the sampled measurement,
- capture, using the first camera, a third video stream after determining the increased priority state,
- capture, using the second camera, a fourth video stream substantially concurrent with the capture of the third video stream,
- process the third video stream to generate a third processed video stream based on the increased priority state, wherein a first processed video stream bitrate is not less than a third processed video stream bitrate,
- process the fourth video stream to generate a fourth processed video stream based on the increased priority state, wherein a second processed video stream bitrate is less than a fourth processed video stream bitrate,
- combine the third and fourth processed video streams into a second combined video stream, and
- transmit the second combined video stream to the remote operation system.
11 more elements
A method for remote operation of a robot, comprising:
- at the robot:
- processing a first video stream captured using a first camera at the robot to generate a first processed video stream,
- processing a second video stream, captured substantially concurrent with the first video stream using a second camera at the robot, to generate a second processed video stream, comprising performing a bitrate reduction process,
- combining the first and second processed video streams into a first combined video stream,
- transmitting the first combined video stream to a remote operation system, said transmitting including transmitting the first combined video stream and a redundant copy of the first combined video stream to the remote operation system for selection by the remote operation system,
- in response to receiving a control instruction from the remote operation system after said transmitting the first combined video stream and the redundant copy of the first combined video stream to the remote operation system, controlling robot operation based on the control instruction,
- after controlling robot operation based on the control instruction, sampling a measurement,
- based on the measurement, determining an increased priority state associated with the second camera,
- based on the increased priority state, processing a third video stream captured using the first camera to generate a third processed video stream, wherein a first processed video stream bitrate is not less than a third processed video stream bitrate,
- based on the increased priority state, processing a fourth video stream, captured using the second camera substantially concurrent with the capturing of the third video stream, to generate a fourth processed video stream, wherein a second processed video stream bitrate is less than a fourth processed video stream bitrate, and
- combining the third and fourth processed video streams into a second combined video stream.
8 more elements
Dependencies are inferred from the claim text.