Projected augmented reality interface with pose tracking for directing manual processes
OwnerDag Michael Peter Hanssoninventor
Life of this patent
Estimating the term…
Abstract
Technologies are described for providing a projected augmented reality system with pose tracking for directing manual processes. A projected augmented reality system includes a video projector, configured to project a dynamically-changing image onto a surface within a work area of an operator, a sensor, and a computer. The computer includes a memory including instructions that when executed cause the computer to obtain three-dimensional pose data using the sensor, determine an output graphical element based on a sequence of three-dimensional pose data over time, and on a current production state, and send an output image based on the output graphical element to the video projector for projection onto the surface.
Claims40
A projected augmented reality system for directing an operator to perform a manual process, comprising:
- a video projector, configured to project a dynamically-changing image onto a surface within a work area of the operator;
- a two-dimensional camera with a wide field of view configured to capture two-dimensional images of a large area including the operator and of the work area;
- a two-dimensional camera with a narrow field of view configured to capture more-detailed two-dimensional images of a part within the large area;
- a depth sensor configured to capture three-dimensional images of the operator and of the work area;
- and
- a computer communicatively connected to the video projector, the two-dimensional camera with a wide field of view, the two-dimensional camera with a narrow field of view, and the depth sensor, the computer comprising a memory and a processor, the memory including instructions that when executed by the processor cause the computer to:
- receive a two-dimensional image of the operator and of at least a portion of the work area from the two-dimensional camera with a wide field of view;
- determine pose data for the operator from the two-dimensional image;
- receive a three-dimensional image of the operator and of the work area from the depth sensor;
- update the pose data with depth coordinates based on the three-dimensional image to provide three-dimensional pose data;
- receive a more detailed two-dimensional image of a part within the large area from the two-dimensional camera with a narrow field of view;
- determine a part and part placement data for the operator from the two-dimensional image;
- determine an output graphical image based on the sequence of three-dimensional pose data over time, the part and part placement data, and on the current production state;
- and
- send the output graphical image to the video projector for projection onto the work area.
A method for using a projected augmented reality system for directing an operator to perform a manual process, comprising:
- receiving by a computer a wide field of view two-dimensional image of the operator and of at least a portion of a work area from an image sensor;
- determining by the computer pose data for the operator from the wide field of view two-dimensional image;
- receiving by the computer a three-dimensional image of the operator and of the work area from a depth sensor;
- updating by the computer the pose data with depth coordinates based on the three-dimensional image to provide three-dimensional pose data;
- receiving a more detailed narrow field of view two-dimensional image of a part within the wide field of view two-dimensional image;
- determining a part and part placement data for the operator from the narrow field of view two-dimensional image;
- determining an output graphical image based on the three-dimensional pose data over time, the part and part placement data, and on a current production state;
- and
- sending the output graphical image onto a surface within the work area using a video projector.
A projected augmented reality system for directing an operator to perform a manual process, comprising:
- a video projector, configured to project a dynamically-changing image onto a surface within a work area of the operator;
- a depth sensor;
- a two-dimensional camera with a narrow field of view;
- and
- a computer communicatively connected to the video projector and the depth sensor, the computer comprising a memory and a processor, the memory including instructions that when executed by the processor cause the computer to:
- obtain three-dimensional pose data using the depth sensor;
- obtain part, part placement and part quality data using the two-dimensional camera with a narrow field of view;
- determine an output graphical image based on the part, part placement and part quality data, and on a current production state;
- and
- sending an output image based on the output graphical image onto a surface within the work area using a video projector.
Dependencies are inferred from the claim text.