Systems and methods for operating a robotic system and executing robotic interactions

A robotic system, comprising:

• one or more hardware processors operable to:
• receive environment data corresponding to a current environment:
• from one or more sensors configured in the robotic system;
• detect one or more objects in the current environment;
• retrieve interaction data corresponding to the one or more objects from a memory associated with the robotic system;
• and
• execute one or more interactions on one or more corresponding objects in the one or more objects, based on the interaction data, wherein executing at least one of the one or more interactions on the one or more corresponding objects in the one or more objects comprises for each or the one or more interactions:
• positioning one or more end effectors within a proximity of the corresponding one or more objects;
• identifying one or more predefined positions of the one or more end effectors relative to the corresponding one or more objects, the predetermined standard position being selected from one or more standard positions of the one or more end effectors;
• positioning the one or more end effectors at the identified standard position using one or more positioning techniques, the one or more positioning techniques including an object template matching technique or a marker-based technique, the object template matching technique having a sensor matching technique for use with standard objects or respective corresponding locations, the marker-based technique for use with the standard objects or non-standard objects;
• and
• controlling the one or more end effectors to execute the one or more interactions on the corresponding one or more objects;
• wherein positioning one or more end effectors at a standard position using the marker-based technique comprises detecting one or more markers associated with a target object;
• and adjusting position of the one or more end effectors towards the standard position based on the detected one or more markers associated with the target object, wherein the position is adjusted using a real-time image of the target object received from at least one image capturing device associated with the one or more end effectors;
• wherein the one or more markers comprises at least one of a physical marker disposed on the target object or a virtual marker corresponding to one or more points on the target object, wherein the one or more markers enable computation of position parameters comprising distance, orientation, angle, or slope, of the one or more end effectors with respect to the target object;
• wherein the virtual markers are identified on the target object using at least one of a plurality of techniques:
• shape analysis technique, particle filtering technique or Convolutional Neural Network (CNN) technique;
• and
• wherein identifying the virtual markers using the CNN technique comprises executing a CNN model corresponding to the target object from one or more libraries stored in the memory associated with the robotic system;
• and detecting positions on the target object for positioning the virtual markers using the CNN model.

A method for operating a robotic system, the robotic system having one or more robotic arms coupled to one or more robotic end effectors, comprising:

• receiving, by one or more processors in a robotic system, environment data corresponding to a current environment, from one or more sensors;
• detecting, by the one or more processors, one or more objects in the current environment;
• and
• retrieving, by the one or more processors, one or more interaction data corresponding to each of the one or more objects from a memory associated with the robotic system;
• executing, by the one or more processors, one or more interactions on one or more corresponding objects in the one or more objects, based on the interaction data, wherein executing at least one of the one or more interactions on the one or more corresponding objects in the one or more objects comprises for each of the one or more interactions:
• positioning one or more end effectors within a proximity of the corresponding one or more objects;
• identifying one or more predefined positions of the one or more end effectors relative to the corresponding one or more objects, the predetermined standard position being selected from one or more standard positions of the one or more end effectors;
• positioning the one or more end effectors at the identified standard position using one or more positioning techniques, the one or more positioning techniques including an object template matching technique or a marker-based technique, the object template matching technique having a sensor matching technique for use with standard objects or respective corresponding locations, the marker-based technique for use with the standard objects or non-standard objects;
• and
• controlling the one or more end effectors to execute the one or more interactions on the corresponding one or more objects;
• wherein positioning one or more end effectors at a standard position using the marker-based technique comprises detecting one or more markers associated with a target object;
• and adjusting position of the one or more end effectors towards the standard position based on the detected one or more markers associated with the target object, wherein the position is adjusted using a real-time image of the target object received from at least one image capturing device associated with the one or more end effectors;
• wherein the one or more markers comprises at least one of a physical marker disposed on the target object or a virtual marker corresponding to one or more points on the target object, wherein the one or more markers enable computation of position parameters comprising distance, orientation, angle, or slope, of the one or more end effectors with respect to the target object;
• wherein the virtual markers are identified on the target object using at least one of a plurality of techniques:
• shape analysis technique, particle filtering technique or Convolutional Neural Network (CNN) technique;
• and
• wherein identifying the virtual markers using shape analysis technique comprises:
• receiving real-time images of the target object from at least one image capturing device associated one or more end effectors;
• determining shape of the target object and longest and shortest sides of the target object, wherein sides of the target object are determined as longest and shortest with reference to length of each side of the target object;
• determining geometric center of the target object based on the shape of the target object and, the longest and the shortest sides of the target object;
• and
• positioning a geometric shape on the target object, each side of the geometric shape is equal to half of the shortest side of the target object;
• the geometric shape oriented along the longest side of the target object;
• and the geometric shape having a geometric center that coincides with a geometric center of the target object;
• and positioning by the virtual markers at each vertex of the geometric shape.

A method for operating a robotic system, the robotic system having one or more robotic arms coupled to one or more robotic end effectors, comprising:

• receiving, by one or more processors in a robotic system, environment data corresponding to a current environment, from one or more sensors;
• detecting, by the one or more processors, one or more objects in the current environment;
• and
• retrieving, by the one or more processors, one or more interaction data corresponding to each of the one or more objects from a memory associated with the robotic system;
• executing, by the one or more processors, one or more interactions on one or more corresponding objects in the one or more objects, based on the interaction data, wherein executing at least one of the one or more interactions on the one or more corresponding objects in the one or more objects comprises for each of the one or more interactions:
• positioning the one or more end effectors within a proximity of the corresponding one or more objects;
• identifying one or more predefined positions of the one or more end effectors relative to the corresponding one or more objects, the predetermined standard position being selected from one or more standard positions of the one or more end effectors;
• positioning the one or more end effectors at the identified standard position using one or more positioning techniques, the one or more positioning techniques including an object template matching technique or a marker-based technique, the object template matching technique having a sensor matching technique for use with standard objects or respective corresponding locations, the marker-based technique for use with the standard objects or non-standard objects;
• and
• controlling the one or more end effectors to execute the one or more interactions on the corresponding one or more objects;
• wherein positioning one or more end effectors at a standard position using the marker-based technique comprises detecting one or more markers associated with a target object;
• and adjusting position of the one or more end effectors towards the standard position based on the detected one or more markers associated with the target object, wherein the position is adjusted using a real-time image of the target object received from at least one image capturing device associated with the one or more end effectors;
• wherein the one or more markers comprises at least one of a physical marker disposed on the target object or a virtual marker corresponding to one or more points on the target object, wherein the one or more markers enable computation of position parameters comprising distance, orientation, angle, or slope, of the one or more end effectors with respect to the target object;
• wherein the virtual markers are identified on the target object using at least one of a plurality of techniques:
• shape analysis technique, particle filtering technique or Convolutional Neural Network (CNN) technique;
• and
• wherein positioning the virtual markers using particle filtering technique comprises:
• retrieving one or more predetermined values corresponding to predetermined positions of the target object from a memory associated with the robotic system;
• receiving real-time images of the target object from at least one image capturing device associated with the one or more end effectors;
• generating one or more points within boundaries of the target object using the real-time images;
• determining an estimated value for combination of visual features in neighborhood of each point, wherein the visual features comprises at least one of histograms of gradients, spatial color distributions or texture features;
• comparing each estimated value with each of the one or more predetermined values to identify respective proximal match;
• and
• positioning the virtual markers at each position on the target object corresponding to each proximal match.

A method for operating a robotic system, the robotic system having one or more robotic arms coupled to one or more robotic end effectors, comprising:

• receiving, by one or more processors in a robotic system, environment data corresponding to a current environment, from one or more sensors;
• detecting, by the one or more processors, one or more objects in the current environment;
• and
• retrieving, by the one or more processors, one or more interaction data corresponding to each of the one or more objects from a memory associated with the robotic system;
• executing, by the one or more processors, one or more interactions on one or more corresponding objects in the one or more objects, based on the interaction data, wherein executing at least one of the one or more interactions on the one or more corresponding objects in the one or more objects comprises for each of the one or more interactions:
• positioning one or more end effectors within a proximity of the corresponding one or more objects;
• identifying one or more predefined positions of the one or more end effectors relative to the corresponding one or more objects, the predetermined standard position being selected from one or more standard positions of the one or more end effectors;
• positioning the one or more end effectors at the identified standard position using one or more positioning techniques, the one or more positioning techniques including an object template matching technique or a marker-based technique, the object template matching technique having a sensor matching technique for use with standard objects or respective corresponding locations, the marker-based technique for use with the standard objects or non-standard objects;
• and
• controlling the one or more end effectors to execute the one or more interactions on the corresponding one or more objects;
• wherein positioning one or more end effectors at a standard position using the marker-based technique comprises detecting one or more markers associated with a target object;
• and adjusting position of the one or more end effectors towards the standard position based on the detected one or more markers associated with the target object, wherein the position is adjusted using a real-time image of the target object received from at least one image capturing device associated with the one or more end effectors;
• wherein the one or more markers comprises at least one of a physical marker disposed on the target object or a virtual marker corresponding to one or more points on the target object, wherein the one or more markers enable computation of position parameters comprising distance, orientation, angle, or slope, of the one or more end effectors with respect to the target object;
• and
• wherein the one or more markers include the physical marker disposed on the target object, wherein the physical marker is a geometric shape marker, and wherein adjusting position of the one or more end effectors comprises:
• moving the one or more end effectors towards the geometric shape marker until at least one side of the geometric shape marker has a preferred length;
• rotating the one or more end effectors until a bottom vertex of the geometric shape marker is disposed in a bottom position of the real-time image of the target object;
• shifting the one or more end effectors along an x-axis or y-axis of the real-time image of the target object until a center of the geometric shape marker is in a center position of the real-time image of the target object;
• and
• adjusting a slope of the one or more end effectors until each angle of the geometric shape marker are at least one of equal to a predetermined maximum difference between the angles that is smaller than their difference prior to initiating the adjustment of the position of the one or more end effectors.

A robotic system, comprising:

• one or more hardware processors operable to:
• receive environment data corresponding to a current environment from one or more sensors configured in the robotic system;
• detect one or more objects in the current environment;
• retrieve interaction data corresponding to the one or more objects from a memory associated with the robotic system;
• and
• execute one or more interactions on one or more corresponding objects in the one or more objects, based on the interaction data, wherein executing at least one of the one or more interactions on the one or more corresponding objects in the one or more objects comprises for each of the one or more interactions:
• positioning one or more end effectors within a proximity of the corresponding one or more objects;
• identifying one or more predefined positions of the one or more end effectors relative to the corresponding one or more objects, the predetermined standard position being selected from one or more standard positions of the one or more end effectors;
• positioning the one or more end effectors at the identified standard position using one or more positioning techniques, the one or more positioning techniques including an object template matching technique or a marker-based technique, the object template matching technique having a sensor matching technique for use with standard objects or respective corresponding locations, the marker-based technique for use with the standard objects or non-standard objects;
• and
• controlling the one or more end effectors to execute the one or more interactions on the corresponding one or more objects;
• wherein positioning one or more end effectors at a predetermined standard position using the marker-based technique comprises detecting one or more markers associated with a target object;
• and adjusting position of the one or more end effectors towards the standard position based on the detected one or more markers associated with the target object, wherein the position is adjusted using a real-time image of the target object received from at least one image capturing device associated with the one or more end effectors;
• wherein the one or more markers comprises at least one of a physical marker disposed on the target object or a virtual marker corresponding to one or more points on the target object, wherein the one or more markers enable computation of position parameters comprising distance, orientation, angle, or slope, of the one or more end effectors with respect to the target object;
• wherein the virtual markers are identified on the target object using at least one of a plurality of techniques:
• shape analysis technique, particle filtering technique or Convolutional Neural Network (CNN) technique;
• and
• wherein positioning the virtual markers using shape analysis technique by:
• receiving real-time images of a target object from at least one image capturing device associated with the one or more end effectors;
• determining a shape of the target object and longest and shortest sides of the target object, wherein the sides of the target object are determined as longest and shortest with reference to length of each side of the target object;
• determining a geometric center of the target object based on the shape of the target object, and the longest and the shortest sides of the target object;
• and
• positioning a geometric shape on the target object, wherein each side of the geometric shape is equal to a portion of the shortest side of the target object;
• the geometric shape is oriented along the longest side of the target object;
• and the geometric shape having a geometric shape that coincides with the geometric center of the target object;
• and positioning the virtual markers at each vertex of the geometric shape.

A robotic system, comprising:

• one or more hardware processors operable to:
• receive environment data corresponding to a current environment from one or more sensors configured in the robotic system;
• detect one or more objects in the current environment;
• retrieve interaction data corresponding to the one or more objects from a memory associated with the robotic system;
• and
• execute one or more interactions on one or more corresponding objects in the one or more objects, based on the interaction data, wherein executing at least one of the one or more interactions on the one or more corresponding objects in the one or more objects comprises for each of the one or more interactions:
• positioning one or more end effectors within a proximity of the corresponding one or more objects;
• identifying one or more predefined positions of the one or more end effectors relative to the corresponding one or more objects, the predetermined standard position being selected from one or more standard positions of the one or more end effectors;
• positioning the one or more end effectors at the identified standard position using one or more positioning techniques, the one or more positioning techniques including an object template matching technique or a marker-based technique, the object template matching technique having a sensor matching technique for use with standard objects or respective corresponding locations, the marker-based technique for use with the standard objects or non-standard objects;
• and
• controlling the one or more end effectors to execute the one or more interactions on the corresponding one or more objects;
• wherein positioning one or more end effectors at a predetermined standard position using the marker-based technique comprises detecting one or more markers associated with a target object;
• and adjusting position of the one or more end effectors towards the standard position based on the detected one or more markers associated with the target object, wherein the position is adjusted using a real-time image of the target object received from at least one image capturing device associated with the one or more end effectors;
• wherein the one or more markers comprises at least one of a physical marker disposed on the target object or a virtual marker corresponding to one or more points on the target object, wherein the one or more markers enable computation of position parameters comprising distance, orientation, angle, or slope, of the one or more end effectors with respect to the target object;
• wherein the virtual markers are identified on the target object using at least one of a plurality of techniques:
• shape analysis technique, particle filtering technique or Convolutional Neural Network (CNN) technique;
• and
• wherein positioning the virtual markers using particle filtering technique by:
• retrieving one or more predetermined values corresponding to predetermined positions of the target object from a memory associated with the robotic system;
• receiving real-time images of the target object from at least one image capturing device associated with one or more end effectors;
• generating one or more points within boundaries of the target object using the real-time images;
• determining an estimated value for combination of visual features in neighborhood of each point in the one more points, wherein the visual features comprises at least one of histograms of gradients, spatial color distributions or texture features;
• comparing each estimated value with each of the one or more predetermined values to identify respective proximal match;
• and
• positioning the virtual markers at each position on the target object corresponding to each proximal match.

A robotic system, comprising:

• one or more hardware processors operable to:
• receive environment data corresponding to a current environment from one or more sensors configured in the robotic system;
• detect one or more objects in the current environment;
• retrieve interaction data corresponding to the one or more objects from a memory associated with the robotic system;
• and
• execute one or more interactions on one or more corresponding objects in the one or more objects, based on the interaction data, wherein executing at least one of the one or more interactions on the one or more corresponding objects in the one or more objects comprises for each of the one or more interactions:
• positioning one or more end effectors within a proximity of the corresponding one or more objects;
• identifying one or more predefined positions of the one or more end effectors relative to the corresponding one or more objects, the predetermined standard position being selected from one or more standard positions of the one or more end effectors;
• positioning the one or more end effectors at the identified standard position using one or more positioning techniques, the one or more positioning techniques including an object template matching technique or a marker-based technique, the object template matching technique having a sensor matching technique for use with standard objects or respective corresponding locations, the marker-based technique for use with the standard objects or non-standard objects;
• and
• controlling the one or more end effectors to execute the one or more interactions on the corresponding one or more objects;
• wherein positioning one or more end effectors at a predetermined standard position using the marker-based technique comprises detecting one or more markers associated with a target object;
• and adjusting position of the one or more end effectors towards the standard position based on the detected one or more markers associated with the target object, wherein the position is adjusted using a real-time image of the target object received from at least one image capturing device associated with the one or more end effectors;
• wherein the one or more markers comprises at least one of a physical marker disposed on the target object or a virtual marker corresponding to one or more points on the target object, wherein the one or more markers enable computation of position parameters comprising distance, orientation, angle, or slope, of the one or more end effectors with respect to the target object;
• wherein the one or more markers include the physical marker disposed on the target object, wherein the physical marker is a geometric shape marker, and wherein the one or more processors adjust position of the one or more end effectors by:
• moving the one or more end effectors towards the geometric shape marker until at least one side of the geometric shape marker has a preferred length;
• rotating the one or more end effectors until a bottom vertex of the geometric shape marker is disposed in a bottom position of the real-time image of the target object;
• shifting the one or more end effectors along an x-axis or y-axis of the real-time image of the target object until a center of the geometric shape marker is in or near a center position of the real-time image of the target object;
• and
• adjusting a slope of the one or more end effectors until each angle of the triangle-shaped marker is equal to a predetermined maximum difference between the angles that is smaller than their difference prior to initiating the adjustment of the position of the one or more end effectors.