# US 12,008,768 B2 · Spatially-aware camera and method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,768 B2 - Document ID: US12008768B2 - Status: Granted - Kind: B2 - Application number: 17/055,108 - Filed: 2019-09-24 - Published: 2021-09-02 - Granted: 2024-06-11 - Earliest claimed: 2018-09-25 - Assignees: Visual Robotics Systems Inc. - Applicants: Visual Robotics Systems Inc. - Inventors: Bryan Lee Olmstead - Web page: https://patentgrep.com/patent/US12008768B2 ## CPC - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - B25J 9/1697: {Vision controlled systems} - G06T 7/55: from multiple images - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06T 2207/30244: Camera pose - G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking - G06V 2201/034: of medical instruments - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Mobile apparatus (e.g., robotic systems and mobile vehicles) use one or more cameras fixed to the apparatus so that the cameras move when the mobile apparatus moves. The cameras acquire images of the world space in which the apparatus operates, and the images are processed to develop information about objects sharing the world space with the mobile apparatus. This information permits a control system to direct the apparatus to interact with (or avoid) the objects. ## Claims ### Claim 1 A mobile apparatus, comprising: a camera system mounted to the mobile apparatus, comprising a camera, the camera comprising a clock; a programmable control system coupled to the mobile apparatus and to the camera, containing data and instructions to cause the programmable control system to: receive from the camera a plurality of images of an object taken at respective successive exposure times while the object is in motion; obtain from the clock a corresponding plurality of timestamps for the respective successive exposure times; receive from the mobile apparatus a plurality of location-based data for the mobile apparatus at or around the respective successive exposure times and associate the received location-based data for the mobile apparatus at or around the respective successive exposure times with the plurality of timestamps for the respective successive exposure times; estimate a plurality of locations of the camera at the respective successive exposure times based on the plurality of location-based data for the mobile apparatus at or around the exposure times associated with the plurality of timestamps for the respective successive exposure times; analyze the plurality of images of the object taken at the respective successive exposure times and the estimated plurality of locations of the camera at the respective successive exposure times associated with the plurality of timestamps for the respective successive exposure times and determine therefrom a location of the object and an accurate three-dimensional model representing the object; and instruct the mobile apparatus to move with respect to the location of the object, responsive to the analysis. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the data and instructions to cause the programmable control system to receive from the camera the plurality of images of the object taken at respective successive exposure times while the object is in motion, comprises data and instructions to cause the programmable control system to receive from the camera a plurality of two-dimensional images of the object taken at respective successive exposure times while the object is in motion; wherein the further data and instructions to cause the programmable control system to analyze the plurality of images of the object taken at respective successive exposure times while the object is in motion and determine therefrom a location of the object, comprises data and instructions to cause the programmable control system to: analyze the two-dimensional images of the object taken at the respective successive exposure times while the object is in motion and determine therefrom a plurality of locations of two-dimensional features of the object; and determine, via photogrammetry, parameters of the accurate three-dimensional model representing the object based on the plurality of locations of two-dimensional features of the object. ### Claim 3 (depends on 2) The apparatus of claim 2, wherein the parameters of the accurate three-dimensional model representing the object are selected from a group of parameters consisting of: a location, an orientation, a size, a velocity, and an acceleration, of the object. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein the mobile apparatus further comprises an effector by which to manipulate the object; and wherein the data and instructions are further configured to cause the programmable control system to perform an operation with respect to the object, the operation selected from a group of operations consisting of: moving the effector with respect to the object, and making contact between the effector and the object, based on the accurate three-dimensional model representing the object. ### Claim 5 A mobile apparatus, comprising: a camera system mounted to the mobile apparatus, comprising a camera, the camera comprising a clock; a programmable control system coupled to the mobile apparatus and to the camera, containing data and instructions to cause the programmable control system to: receive from the camera a plurality of two-dimensional images of an object taken at respective successive exposure times; obtain from the clock a plurality of timestamps corresponding to the respective successive exposure times; receive from the mobile apparatus a plurality of locations for the mobile apparatus at or around the respective successive exposure times and associate the received plurality of locations for the mobile apparatus at or around the respective successive exposure times with the plurality of timestamps for the respective successive exposure times; estimate a plurality of locations of the camera at the respective successive exposure times based on the plurality of locations for the mobile apparatus at or around the respective successive exposure times associated with the plurality of timestamps for the respective successive exposure times; and analyze the plurality of two-dimensional images of the object taken at the respective successive exposure times and the estimated plurality of locations of the camera at the respective successive exposure times associated with the plurality of timestamps for the respective successive exposure times and determine therefrom a plurality of locations of two-dimensional features of the object; determine, via photogrammetry, parameters of an accurate three-dimensional model representing the object based on the plurality of locations of two-dimensional features of the object; and instruct the mobile apparatus to move with respect to a location of the object, responsive to the analysis. ### Claim 6 (depends on 5) The apparatus of claim 5, wherein the mobile apparatus further comprises an effector by which to manipulate the object; and wherein the data and instructions are further configured to cause the programmable control system to perform an operation with respect to the object, the operation selected from a group of operations consisting of: moving the effector with respect to the object, and making contact between the effector and the object, based on the accurate three-dimensional model representing the object. ### Claim 7 (depends on 5) The apparatus of claim 5, wherein the parameters of the three-dimensional model representing the object are selected from a group of parameters consisting of: a location, an orientation, a size, a velocity, and an acceleration, of the object. ### Claim 8 (depends on 5) The apparatus of claim 5, wherein the data and instructions to cause the programmable control system to receive from the camera the plurality of two-dimensional images of the object taken at respective successive exposure times, comprises data and instructions to cause the programmable control system to receive from the camera the plurality of two-dimensional images of the object taken at respective successive exposure times while the mobile apparatus is in motion. ### Claim 9 (depends on 5) The apparatus of claim 5, wherein the data and instructions to cause the programmable control system to receive from the camera the plurality of two-dimensional images of the object taken at respective successive exposure times, comprises data and instructions to cause the programmable control system to receive from the camera the plurality of two-dimensional images of the object taken at respective successive exposure times while the object is in motion. ### Claim 10 (depends on 9) The mobile apparatus of claim 9, wherein the data and instructions to cause the programmable control system to estimate the plurality of locations of the camera at the respective successive exposure times based on the plurality of locations for the mobile apparatus at or around the respective successive exposure times associated with the plurality of timestamps for the respective successive exposure times, comprises data and instructions to cause the programmable control system to interpolate the location of the camera at an exposure time associated with a timestamp for the exposure time based on a first set of coordinates of a first location of the mobile apparatus at a first time before the exposure time and a second set of coordinates of a second location of the mobile apparatus at a second time after the exposure time. ### Claim 11 (depends on 9) The mobile apparatus of claim 9, wherein the camera system further comprises an inertial measurement unit (IMU) that measures velocity and/or acceleration of the camera; wherein the data and instructions further cause the programmable control system to receive measured velocity and/or acceleration of the camera from the IMU; and wherein the data and instructions that cause the programmable control system to estimate the plurality of locations of the camera at the respective successive exposure times based on the plurality of locations for the mobile apparatus at or around the respective successive exposure times associated with the plurality of timestamps for the respective successive exposure times, comprises data and instructions that cause the programmable control system to estimate a location of the camera at an exposure time associated with a timestamp for the exposure time based on a location for the mobile apparatus at or around the exposure time associated with the timestamp for the exposure time and the received measured velocity and/or acceleration of the camera from the IMU. ### Claim 12 (depends on 11) The mobile apparatus of claim 11, wherein the data and instructions that cause the programmable control system to estimate the location of the camera at the exposure time associated with the timestamp for the exposure time based on the location for the mobile apparatus at or around the exposure time associated with the timestamp for the exposure time and the received measured velocity and/or acceleration of the camera from the IMU, comprises data and instructions that cause the programmable control system to extrapolate the location of the camera at the exposure time associated with the timestamp for the exposure time based on a set of coordinates of a location of the mobile apparatus either at a time before or after the exposure time.