# US 11,027,425 B1 · Space extrapolation for robot task performance > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,027,425 B1 - Document ID: US11027425B1 - Status: Granted - Kind: B1 - Application number: 16/041,164 - Filed: 2018-07-20 - Granted: 2021-06-08 - Earliest claimed: 2016-06-16 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Crystal Chao - Web page: https://patentgrep.com/patent/US11027425B1 ## CPC - B25J 9/1661: {characterised by task planning, object-oriented languages} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1697: {Vision controlled systems} - B25J 19/021: {Optical sensing devices} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - B25J 9/1664: {characterised by motion, path, trajectory planning} - B25J 9/1679: {characterised by the tasks executed} - B25J 19/023: {including video camera means} - G05B 2219/39466: Hand, gripper, end effector of manipulator - G05B 2219/40014: Gripping workpiece to place it in another place - G05B 2219/40116: Learn by operator observation, symbiosis, show, watch - G05B 2219/45063: Pick and place manipulator - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Methods, apparatus, systems, and computer-readable media are provided for enabling users to approximately identify a space within an environment inhabited by a plurality of objects that user wishes for a robot to manipulate. In various implementations, an approximation of a space within an environment may be identified based on user input. The actual space within the environment may then be extrapolated based at least in part on the approximation and one or more attributes of the environment. A plurality of objects that are co-present within the space and that are to be manipulated by a robot may be identified. The robot may then be operated to manipulate the identified plurality of objects. ## Claims ### Claim 1 A computer-implemented method, comprising: identifying, by one or more processors, based on a gesture of an operator detected in vision data captured by one or more vision sensors, an approximation of a discrete space within the environment, wherein identifying the approximation comprises projecting one or more vectors from the gesture to one or more points on a surface in the environment; extrapolating, by one or more of the processors, the discrete space within the environment based on the approximation and one or more visual attributes of the environment captured by one or more of the vision sensors; identifying, by one or more of the processors, a plurality of objects that are co-present within the discrete space, wherein the plurality of objects are to be manipulated by a robot; and operating, by one or more of the processors, the robot to manipulate the identified plurality of objects. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein identifying the approximation comprises temporally synchronizing user input with audio data sensed by one or more audio sensors. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising identifying one or more frames of the vision data that correspond temporally with one or more utterances captured in audio data that are associated with user identification of space. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, further comprising performing, by one or more of the processors, semantic processing of the audio data to associate the one or more utterances with user identification of space. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising identifying, by one or more of the processors, based on the approximation, at least one bounded physical surface in the environment that is associated with the discrete space. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising: detecting, by one or more of the processors, one or more physical edges of the at least one bounded physical surface; and defining, by one or more of the processors, one or more boundaries of the discrete space based at least in part on the one or more detected physical edges. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein detecting the one or edges comprises performing edge detection on visual data sensed by one or more of the vision sensors. ### Claim 8 (depends on 6) The computer-implemented method of claim 6, wherein detecting the one or edges is based on one or more surfaces identified within a three-dimensional model of the environment available to the one or more processors. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the surface is a horizontal surface. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the surface is a vertical surface to which the plurality of objects are secured or mounted. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the plurality of objects are heterogeneous. ### Claim 12 A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to: identify, based on a gesture of an operator detected in vision data captured by one or more vision sensors, an approximation of a discrete space within an environment, wherein the approximation is identified by projecting a plurality of vectors to a plurality of points on a surface in the environment; define, based on the plurality of points, the discrete space within the environment; identify a plurality of objects that are co-present within the discrete space, wherein the plurality of objects are to be manipulated by a robot; and operate the robot to manipulate the identified plurality of objects. ### Claim 13 (depends on 12) The system of claim 12, further comprising instructions to temporally synchronize user input with audio data sensed by one or more audio sensors. ### Claim 14 (depends on 13) The system of claim 13, wherein the operator gesture is detected in the visual data based at least in part by identifying one or more frames of the vision data that correspond temporally with one or more utterances captured in the audio data that are associated with user identification of space. ### Claim 15 A method implemented using one or more processors, comprising: identifying, based on a gesture of an operator detected in vision data captured by one or more vision sensors, an approximation of a discrete space within an environment, wherein the approximation is identified by projecting a plurality of vectors to a plurality of points on a surface in the environment; defining, based on the plurality of points, the discrete space within the environment; identifying a plurality of objects that are co-present within the discrete space, wherein the plurality of objects are to be manipulated by a robot; and operating the robot to manipulate the identified plurality of objects.