Hybrid optical-electronic system for on-edge multimodal and multitask visual intelligence

A computer-implemented method comprising:

• extracting, using an optical metasurface comprising a plurality of meta-atoms, global-image features associated with an image, wherein the global-image features are extracted based on a first set of optical characteristics associated with the image, and wherein the first set of optical characteristics are captured based on a first diffraction distance between the optical metasurface and an image sensor;
• and
• extracting, using the optical metasurface, local-region features associated a region of the image, wherein the local-region features are extracted based on a second set of optical characteristics associated with the region of the image, wherein the second set of optical characteristics are captured based on a second diffraction distance between the optical metasurface and the image sensor, and wherein the first diffraction distance is greater than the second diffraction distance;
• accessing, using an image sensor, the global-image features of the image and the local-region features of the region of the image;
• processing the global-image features and the local-region features using an electronic neural network to generate a set of machine-learning outputs, wherein the electronic neural network:
• (i) fuses the global-image features and the local-region features into one or more feature vectors;
• and (ii) processes the one or more feature vectors to simultaneously perform different types of image-processing operations, thereby generating the set of machine-learning outputs;
• and
• operating an embedded device based on the set of machine-learning outputs.

A non-transitory, computer-readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to perform operations comprising:

• extracting, using an optical metasurface comprising a plurality of meta-atoms, global-image features associated with an image, wherein the global-image features are extracted based on a first set of optical characteristics associated with the image, and wherein the first set of optical characteristics are captured based on a first diffraction distance between the optical metasurface and an image sensor;
• and
• extracting, using the optical metasurface, local-region features associated a region of the image, wherein the local-region features are extracted based on a second set of optical characteristics associated with the region of the image, wherein the second set of optical characteristics are captured based on a second diffraction distance between the optical metasurface and the image sensor, and wherein the first diffraction distance is greater than the second diffraction distance;
• accessing, using an image sensor, the global-image features of the image and the local-region features of the region of the image;
• processing the global-image features and the local-region features using an electronic neural network to generate a set of machine-learning outputs, wherein the electronic neural network:
• (i) fuses the global-image features and the local-region features into one or more feature vectors;
• and (ii) processes the one or more feature vectors to simultaneously perform different types of image-processing operations, thereby generating the set of machine-learning outputs;
• and
• operating an embedded device based on the set of machine-learning outputs.