Early fusion of neural ray graph networks for multi-view camera setups

An apparatus, comprising:

• a memory storing processor-readable code;
• and
• at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including:
• receiving a plurality of image frames;
• determining an ordered set of neural rays based on the plurality of image frames, wherein each neural ray of the ordered set of neural rays represents three-dimensional positions of pixels of an image frame of the plurality of image frames;
• determining a graph network that represents each neural ray of the ordered set of neural rays as a sequence of points, wherein each point is associated with a node of a plurality of nodes of the graph network;
• and
• determining, based on determining the graph network, a feature set for processing by a transformer network, wherein the feature set includes features of each of the plurality of image frames.

A non-transitory computer-readable medium storing instructions that, when executed by a processing system that includes one or more processors, cause the processing system to perform operations comprising:

• receiving a plurality of image frames;
• determining an ordered set of neural rays based on the plurality of image frames, wherein each neural ray of the ordered set of neural rays represents three-dimensional positions of pixels of an image frame of the plurality of image frames;
• determining a graph network that represents each neural ray of the ordered set of neural rays as a sequence of points, wherein each point is associated with a node of a plurality of nodes of the graph network;
• and
• determining, based on determining the graph network, a feature set for processing by a transformer network, wherein the feature set includes features of each of the plurality of image frames.

A vehicle, comprising:

• a plurality of cameras that together have a field of view spanning around the vehicle;
• and
• a memory storing processor-readable code;
• and
• at least one processor coupled to the memory, the at least one processor in communication with the plurality of cameras and configured to execute the processor-readable code to cause the at least one processor to perform operations including:
• receiving, from the plurality of cameras, a plurality of image frames;
• determining an ordered set of neural rays based on the plurality of image frames and intrinsic parameters of the plurality of cameras, wherein each neural ray of the ordered set of neural rays represents three-dimensional positions of pixels of an image frame of the plurality of image frames;
• determining a graph network that represents each neural ray of the ordered set of neural rays as a sequence of points, wherein each point is associated with a node of a plurality of nodes of the graph network;
• and
• determining, based on determining the graph network, a feature set for processing by a transformer network, wherein the feature set includes features of each of the plurality of image frames.