# US 10,503,976 B2 · Methods and apparatus for autonomous robotic control > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,976 B2 - Document ID: US10503976B2 - Status: Granted - Kind: B2 - Application number: 15/463,299 - Filed: 2017-03-20 - Published: 2017-07-06 - Granted: 2019-12-10 - Earliest claimed: 2014-03-19 - Assignees: Neurala, Inc. - Applicants: Neurala, Inc. - Inventors: Massimiliano Versace; Anatoly Gorshechnikov - Web page: https://patentgrep.com/patent/US10503976B2 ## CPC - G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs - B25J 9/1697: {Vision controlled systems} - G06N 3/0409: {Adaptive resonance theory [ART] networks} - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06T 7/11: Region-based segmentation - G06T 7/187: involving region growing; involving region merging; involving connected component labelling - G06T 7/194: involving foreground-background segmentation - G06T 7/292: Multi-camera tracking - G06T 7/74: {involving reference images or patches} - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G05B 2219/39082: Collision, real time collision avoidance - G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour - G06T 2207/30244: Camera pose - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Sensory processing of visual, auditory, and other sensor information (e.g., visual imagery, LIDAR, RADAR) is conventionally based on “stovepiped,” or isolated processing, with little interactions between modules. Biological systems, on the other hand, fuse multi-sensory information to identify nearby objects of interest more quickly, more efficiently, and with higher signal-to-noise ratios. Similarly, examples of the OpenSense technology disclosed herein use neurally inspired processing to identify and locate objects in a robot's environment. This enables the robot to navigate its environment more quickly and with lower computational and power requirements. ## Claims ### Claim 1 A method comprising: acquiring a plurality of images of at least a portion of an environment with an image sensor; constructing a spatial shroud fitting a form of an object in a first image of the plurality of images from the first image; identifying the object in the first image based at least in part on the spatial shroud; translating and/or transforming the spatial shroud based at least in part on a change in a position and/or an orientation of the image sensor; determining if the spatial shroud fits the form of the object in a second image in the plurality of images; and in response to determining that the spatial shroud fits the form of the object, identifying a position of the object in the environment based at least in part on the spatial shroud. ### Claim 2 (depends on 1) The method of claim 1, wherein constructing the spatial shroud comprises segmenting the first image. ### Claim 3 (depends on 1) The method of claim 1, wherein identifying the object includes creating a shroud-gated representation of a view of the object. ### Claim 4 (depends on 1) The method of claim 1, further comprising: extracting a feature from the first image; and identifying the object in the first image based at least in part on the feature. ### Claim 5 (depends on 4) The method of claim 4, wherein extracting the feature comprises applying a non-linear transformation to the first image. ### Claim 6 (depends on 4) The method of claim 4, further comprising: identifying similarities between the feature and a plurality of features identified in other images in the plurality of images. ### Claim 7 (depends on 6) The method of claim 6, further comprising: clustering the images in the plurality of images based at least in part on the similarities. ### Claim 8 A system comprising: an image sensor to acquire a plurality of images of at least a portion of an environment; a processor communicably coupled to the image sensor, the processor to: construct a spatial shroud fitting a form of an object in a first image of the plurality of images from the first image; identify the object in the first image based at least in part on the spatial shroud; translate and/or transform the spatial shroud based at least in part on a change in a position and/or an orientation of the image sensor; determine if the spatial shroud fits the form of the object in a second image in the plurality of images; and in response to determining that the spatial shroud fits the form of the object in the second image, identify a position of the object in the environment based at least in part on the spatial shroud. ### Claim 9 (depends on 8) The system of claim 8, wherein the processor is configured to construct the spatial shroud by segmenting the first image. ### Claim 10 (depends on 8) The system of claim 8, wherein the processor is configured to identify the object by creating a shroud-gated representation of a view of the object. ### Claim 11 (depends on 8) The system of claim 8, wherein the processor is further configured to: extract a feature from the first image; and identify the object in the first image based at least in part on the feature. ### Claim 12 (depends on 11) The system of claim 11, wherein the processor is configured to extract the feature by applying a non-linear transformation to the first image. ### Claim 13 (depends on 11) The system of claim 11, wherein the processor is further configured to: identify similarities between the feature and a plurality of features identified in other images in the plurality of images. ### Claim 14 (depends on 13) The system of claim 13, wherein the processor is further configured to: cluster the images in the plurality of images based at least in part on the similarities. ### Claim 15 (depends on 1) The method of claim 1, further comprising: positioning and/or orienting the image sensor with respect to the object based at least in part on the object. ### Claim 16 (depends on 1) The method of claim 1, further comprising: adjusting a field of view of the image sensor based at least in part on the change in the position and/or the orientation of the image sensor. ### Claim 17 (depends on 8) The system of claim 8, further comprising: an actuator, operably coupled to the processor and to the image sensor, to adjust a field of view of the image sensor based at least in part on the object. ### Claim 18 (depends on 17) The system of claim 17, wherein the actuator is further configured to adjust the field of view of the image sensor based at least in part on the change in the position and/or the orientation of the image sensor. ### Claim 19 (depends on 1) The method of claim 1, wherein the object is a previously unknown new object. ### Claim 20 (depends on 1) The method of claim 1, further comprising: determining that the spatial shroud does not fit the form of the object in a third image in the plurality of images; and in response to determining that the spatial shroud does not fit the form of the object in the third image, identifying a second object in the third image different from the object; and transmitting instructions to the image sensor to capture images of the second object. ### Claim 21 (depends on 1) The method of claim 1, wherein constructing the spatial shroud occurs without supervision from a user. ### Claim 22 (depends on 1) The method of claim 1, wherein constructing the spatial shroud occurs without manual labeling of the object. ### Claim 23 (depends on 1) The method of claim 1, wherein constructing the spatial shroud occurs without corrupting or forgetting previously learned knowledge.