Robot control based on saliency and invariant spatial representations using hierarchical spatial working memory
AssigneeHRL Laboratories, LLC
InventorsDeepak Khosla, Paul Alex Dow, David Huber
Life of this patent
Estimating the term…
Abstract
A method and apparatus for controlling robots based on prioritized targets extracted from fused visual and auditory saliency maps. The fused visual and auditory saliency map may extend beyond the immediate visual range of the robot yet the methods herein allow the robot to maintain an awareness of targets outside the immediate visual range. The fused saliency map may be derived in a bottom-up or top-down approach and may be feature based or object based.
Claims18
A computer program product for controlling a robot, the computer program product comprising a non-transitory computer-readable medium having computer program instructions stored therein for causing at least one computer to perform operations of:
- controlling the robot to acquire sensor data;
- computing one or more saliency maps from the sensor data based on a robot control plan;
- fusing the one or more saliency maps into one combined saliency map;
- creating a list in hierarchical spatial working memory of one or more targets according to the robot control plan from the combined saliency map;
- prioritizing the targets in the hierarchical spatial working memory according to the robot control plan;
- and
- commanding the robot to focus on the targets in order of priority:
- wherein the hierarchical spatial working memory represents each target as a set of Eye Centered Coordinates, a set of Head Centered Coordinates, a set of Body Centered Coordinates, a set of current Eye Camera positions and a set of current Head positions.
A system for robot control comprising:
- a robot control plan, a first eye camera, a second eye camera, at least a left auditory sensor and a right auditory sensor, all mounted on a robot and all providing data to at least one computer;
- the at least one computer programmed to execute instructions stored on a computer readable medium for causing the computer to perform operations of:
- receive said first eye camera, second eye camera, left and right auditory sensors, upper and lower auditory sensors data;
- compute a visual saliency map from the first eye camera and second eye camera data in the head centered coordinate system;
- compute an auditory saliency map in body centered coordinates from the auditory sensor data;
- fuse the visual and auditory saliency maps according to a weighting function;
- derive a list of targets from the fused saliency map and store them in hierarchical spatial working memory;
- prioritize the list of targets according to the contents of the robot control plan;
- and
- command the robot to focus on the targets in priority order;
- wherein the hierarchical spatial working memory represents each target as a set of Eye Centered Coordinates, a set of Head Centered Coordinates, a set of Body Centered Coordinates, a set of current Eye Camera positions and a set of current Head positions.
A method of controlling a robot comprising acts of:
- receiving visual sensory data;
- receiving auditory sensory data;
- receiving a control plan for the robot;
- calculating a saliency map according to the control plan;
- computing a list of targets in Hierarchical Spatial Working Memory from the saliency map;
- prioritizing the list of targets according to the robot control plan;
- controlling the robot by commanding the robot to focus on the targets in order of priority:
- wherein the Hierarchical Spatial Working Memory represents each target as a set of Eye Centered Coordinates, a set of Head Centered Coordinates, a set of Body Centered Coordinates, a set of current Eye Camera positions and a set of current Head positions.
At least one computer programmed to execute a process for controlling a robot, the process comprising:
- receiving visual sensory data;
- receiving auditory sensory data;
- receiving a control plan for the robot;
- calculating a saliency map according to the control plan;
- computing a list of targets in Hierarchical Spatial Working Memory from the saliency map;
- prioritizing the list of targets according to the robot control plan;
- controlling the robot by commanding the robot to focus in order of priority on the targets:
- wherein the Hierarchical Spatial Working Memory represents each target as a set of Eye Centered Coordinates, a set of Head Centered Coordinates, a set of Body Centered Coordinates, a set of current Eye Camera positions and a set of current Head positions.
Dependencies are inferred from the claim text.