# US 10,766,145 B2 · Eye in-hand robot > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,766,145 B2 - Document ID: US10766145B2 - Status: Granted - Kind: B2 - Application number: 15/953,022 - Filed: 2018-04-13 - Published: 2018-10-18 - Granted: 2020-09-08 - Earliest claimed: 2017-04-14 - Assignees: Brown University - Applicants: Brown University - Inventors: John Oberlin; Stefanie Tellex - Web page: https://patentgrep.com/patent/US10766145B2 ## CPC - B25J 15/0004: {with provision for adjusting the gripped object in the hand} - B25J 9/1607: {Calculation of inertia, jacobian matrixes and inverses} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1697: {Vision controlled systems} - B25J 15/0028: {with movable, e.g. pivoting gripping jaw surfaces} - B25J 19/023: {including video camera means} - B25J 19/04: Viewing devices - G05B 2219/40543: Identification and location, position of components, objects - G05B 2219/40613: Camera, laser scanner on end effector, hand eye manipulator, local - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A robot includes a gripping member configured to move and pick up the object, a camera affixed to the gripping member such that movement of the gripping member causes movement of the camera, the camera configured to measure and store data related to intensity of light and direction of light rays within the environment, an image processing module configured to process the data to generate a probabilistic model defining a location of the object within the environment, and an operation module configured to move the gripping member to the location and pick up the object. ## Claims ### Claim 1 A robot comprising: a gripping member configured to move and pick up an object; a camera affixed to the gripping member such that movement of the gripping member causes movement of the camera, the camera configured to measure and store data related to intensity of light and direction of light rays within the environment; an image processing module configured to process the data to generate a probabilistic model defining a location of the object within an environment; and an operation module configured to move the gripping member to the location and pick up the object. ### Claim 2 (depends on 1) The robot of claim 1 wherein the probabilistic model turns a calibrated eye-in-hand camera into a time-lapse light field camera that can be used for robotic perception. ### Claim 3 (depends on 2) The robot of claim 2 wherein robotic perception comprises observing a scene from multiple perspectives, receiving a sequence of images and a sequence of poses containing camera position and orientation. ### Claim 4 A method for picking up an object within an environment using a robot comprising: moving a camera between a plurality of locations, each location generating a different viewing angle between the camera and the object; measuring data, with respect to the object and the environment, showing intensity of light and direction of light rays in the environment at each location; processing the data to determine a most likely location of the object; and operating a gripper on the robot by moving the gripper to the most likely location and picking up the object with the gripper. ### Claim 5 (depends on 4) The method of claim 4 wherein the step of moving the camera is accomplished by attaching the camera to a mobile member of the robot and moving the mobile member. ### Claim 6 (depends on 4) The method of claim 4 wherein the gripper is attached to the mobile member. ### Claim 7 (depends on 4) The method of claim 4 wherein the step of processing the data to determine a most likely location of the object includes generating a probabilistic model. ### Claim 8 (depends on 4) The method of claim 4 wherein the robot is a modified Baxter robot capable of carrying out the other steps of the method. ### Claim 9 (depends on 4) The method of claim 4 further comprising the step of focusing the camera based on a plurality of known objects within the environment, each known object being located at a known distance from at least one of the locations. ### Claim 10 (depends on 4) The method of claim 4 further comprising the step of identifying reflections using optical cues and modifying the data according to the reflections. ### Claim 11 (depends on 4) The method of claim 4 further comprising the step of calibrating the camera based on a depth associated with the object. ### Claim 12 (depends on 4) The method of claim 4 wherein the object is a non-Lambertian object. ### Claim 13 (depends on 4) The method of claim 4 wherein the environment is a non-Lambertian environment.