Apparatuses and methods for machine vision system including creation of a point cloud model and/or three dimensional model

An autonomous system including a machine vision system comprising:

• at least two cameras with overlapping fields of views (FOV);
• a Camera Settings and Image Capture Module (CSICAPM) that operates a first plurality of machine instructions that operate the at least two cameras and a processor to capture pairs of images from the at least two cameras with known camera settings;
• an Image Calibration Module (ICALM) that receives the pairs of images and generates calibrated two-dimensional (2D) images using geometric camera calibration processing for correcting for lens distortion and relating camera sensor dimensions within each image into three-dimensional (3D) dimensions of 2D objects within each of the pairs of images;
• a Feature Detector and Matching Module that detects predefined pixel patterns associated with same structures in respective said pairs of images then outputs match points (MPs) associated with a detected feature with the predefined pixel patterns;
• first and second category depth estimation (DE) modules (DEM) that generate a plurality of first and second depth estimates for each MP;
• wherein the first category DEM includes:
• a Structure from Motion (SFM) Module to generate an initial first depth estimate and ratios of depth (RoDs) estimate, a depth from defocus (DFD) system that generates relative blur estimates (RBE) for each MP associated with each of said predefined pixel patterns found in each of said pairs of images and adjusts the RoDs estimate;
• a first Polynomial Algorithm (PA) Computation Module (PECSM) that uses a projective camera model based PA to generate data that relates RoDs of MPs, 3D positions of MPs and a projective geometry camera model;
• a Depth from Defocus (DFD) Module that includes a Blur Estimation Module to generate relative blur estimates (RBEs) around MPs within each of the pairs of images;
• a second PA Computation Module that relates a relative blur radius associated with different MPs and a depth coordinate of MPs;
• a first Variable Elimination Module (VEM) that receives PAs generated by the first PECSM and the second PA Computation Module to eliminate some variables and generate a multivariate PA data structure that relates a depth of each MP with the RoDs and RBEs;
• a second VEM that receives a multivariate PA from the first VEM to eliminate variables and generates a single variable PA for each MP;
• an Error and Depth Computation Module (EDCM) that receives the single variable PAs from the second VEM and generates error estimates, 2D images, 3D models, and multiple estimated depths for each of said MPs;
• a Reinforcement Learning Camera Setting Selection Module comprising:
• a State Extraction Module that receives image and system settings and images from the Sensor Data Simulator Module, the ICALM, or the CSICAPM to generate image features and system states;
• a Cost Measurement Module that receives the image features and system states from the State Extraction Module and depth estimation errors and range/variance of depth estimates from the first category DEM to generate a list or table of cost value for state and action pairs;
• and
• an Action Selection Module that receives the image features and system states and list or table of cost value for each state and action pair from the Cost Measurement Module to generate a data structure that stores camera settings or system adjustments;
• wherein the second category DEM comprises a Depth Estimation Neural Network (NN) Training Module and a Trained Real Time Depth Estimation NN Module;
• wherein the Depth Estimation NN Training Module includes:
• the Sensor Data Simulator Module that receives the pairs of images from the CSICAPM or the ICALM and camera settings and system settings from the CSICAPM to generate estimates of images or calibrated corrected 2D images;
• a Deep Convolutional or Auto-Encoder Feature Extraction Layers with Patterned Input Sampling Training Module that receives 2D images from the Sensor Data Simulator Module or the CSICAPM to generate a NN that extracts image features;
• and
• a Depth Prediction Layers Training Module that receives image features from the Deep Convolutional or Auto-Encoder Feature Extraction Layers with Patterned Input Sampling Training Module to generate a trained depth estimator neural network;
• wherein the Trained Real Time Depth Estimation NN Module includes:
• a Deep Convolutional or Auto-Encoder Feature Extraction Layers Module that receives 2D images from the Sensor Data Simulator Module, the ICALM, or the CSICAPM and camera settings and system settings from the CSICAPM and the Deep Convolutional or Auto-Encoder Feature Extraction Layers with Patterned Input Sampling Training Module to generate a data structure of image features;
• and
• a Depth Prediction Layers module that receives image features from the Deep Convolutional or Auto-Encoder Feature Extraction Layers Module and the trained depth estimator neural network from the Depth Prediction Layers Training Module to generate a second category depth estimation and a 3D model of structures.

A method of operating a machine vision system comprising:

• providing the machine vision system comprising first and second cameras with overlapping fields of view (FOV), a processor, memory, data storage system, and a plurality of machine readable instructions adapted to operate the machine vision system stored on the data storage system;
• executing a first plurality of machine instructions stored on the data storage system that capture a plurality of pairs of images from said first and second cameras;
• executing a second plurality of machine instructions stored on the data storage system that performs image feature extraction by searching for predetermined or pre-input image features in the pairs of images and outputs a list of x, y coordinates for point correspondences associated with each predetermined or pre-input image feature;
• executing a third plurality of machine instructions stored on the data storage system that computes a first category depth estimate or z coordinate from each x, y coordinate associated with each point correspondence using a plurality of third operations, wherein said third operations comprise:
• executing a combination of structure from motion (SFM) and depth from defocus (DFD) operations, where the SFM operations are used to respectively determine ratios of depth (RoDs) associated with the first and second cameras, which are associated with a pinhole camera model equation, of a selected set of said predetermined or pre-input image features relative to at least two points of at least one structure within the camera FOVs in a projective geometry model, wherein the DFD operation computes a relative blur estimate (RBE) and ratio of blur to eliminate a calibration step based on a thin lens equation;
• executing a fourth plurality of machine instructions stored on the data storage system that executes a first variable elimination (VE) sequence by simplifying a first system of equations (SoE) defined at least in part by the pinhole camera model equation and substituting RoDs to eliminate two depth variables from the first SoE;
• operating a second VE sequence by simplifying a second SoE defined at least in part by the thin lens equation by substituting RBEs and the RoDs into the second SoE to eliminate another depth variable from the second SoE;
• and operating a third VE sequence by combining results from the first and second VE sequences and inputting them into a Sylvester Resultant matrix and eliminating another depth variable to produce a fourth power SoE which then is used to compute first category depth estimates or z coordinates for each x, and y coordinate associated with each point correspondence;
• and
• executing a fifth plurality of machine instructions stored on the data storage system that computes a second category depth estimate or z coordinate from each x, y coordinate associated with each point correspondence using a depth estimation neural network training system, a depth estimation neural network system operable to produce the second category depth estimate for each x, y coordinate associated with each point correspondence, and a comparator operable to compare said first and second category depth estimates to determine if the two estimates are within a predetermined range from each other, wherein if said first and second category depth estimates are outside of said predetermined range, then said depth estimation neural network training system is operated to produce a revised neural network depth estimator training dataset.

A method for producing a three dimensional model data from multi-perspective two dimensional images comprising:

• capturing a first and second plurality of images from a first and second camera that are oriented with a first and second overlapping fields of view (FOV) with a first controller section;
• determining or setting camera settings comprising focus, exposure, and zoom with a second controller section;
• identifying a plurality of features within two or more sets of the first and second plurality of images using a third control section comprising a feature detection system to identify point correspondences (PCs) in the two or more sets of the first and second plurality of images and store a first and second list of x, y coordinates of PCs associated with each feature found in at least one corresponding pair of said first and second plurality of images;
• performing a first and second depth estimation processing using a fourth controller section respectively comprising performing structure from motion (SFM) processing and depth from defocus (DFD) processing to respectively produce a plurality of depth estimates for each PC and a relative blur estimate around each PC, wherein said DFD processing comprises determining relative blur estimates around said PCs for pairs of points within each of said sets of the first and second plurality of images;
• defining or using a provided first polynomial algorithm (PA) defining or describing a projective camera model using a fifth controller section including respective relationships between x, y coordinates within sets of PCs and said first and second cameras and outputting first and second ratios of depth (RoDs), a list of coefficients for variables in the projective camera model that describe relationships between three dimensional coordinates of arbitrary sets or pairs of combinations of PCs and the first PA;
• defining or using a provided second PA using a sixth controller section describing relationships between PCs and said first and second cameras based on a thin lens equation and computing relative blur ratios using sets of RBEs divided by each other that is used to cancel an element of the second PA;
• executing a simplification processing machine instruction sequence using a seventh controller section to perform variable elimination or reduction of a system of equations (SoE) comprising said first and second PAs comprising a data entry operation comprising substitution of said first and second RoDs into said first PA and second PA to eliminate a first and second variable and output a first simplified SoE (FSSoE) multivariate polynomial;
• solving said FSSoE using an eighth controller section by finding roots of the FSSoE using a system of Sylvester Resultant univariate polynomials to eliminate another variable and generating a second simplified SOE (SSSoE) and multiple possible solutions to the SSSoE;
• determining a best solution for each of said PCs using a ninth controller section by back-substituting each of said possible solutions into each Sylvester Resultant univariate polynomial or the SSSoE and calculating a mean result of the back substitution while excluding outliers of the back substitution to obtain an algebraic or geometric (A/G) error estimate of each depth estimate to produce a first category depth estimate;
• computing and outputting a three-dimensional (3D) model of image structures within said first and second camera overlapping FOVs, wherein said 3D model comprises a second category depth estimate for selected x,y pixels in input two-dimensional (2D) images;
• and
• operating a reinforcement learning camera settings selection module comprising a state extraction module, a cost measurement module, and an action selection module for selecting or determining said camera settings;
• and
• operating a second category depth computation module comprising a monocular depth estimation system comprising a depth estimation neural network (NN) training module and a trained real time depth estimation module which uses a trained NN depth estimation module output from said depth estimation NN training module, wherein said depth estimation NN training module comprises a first deep convolutional or auto-encoder feature extraction layers with patterned input sampling training module and a first depth prediction layer training module configured to compute and output said trained NN depth estimation module, wherein said trained real time depth estimation module comprises a deep convolutional or auto-encoder feature extraction layer with patterned input sampling and a second depth prediction layer module configured to use said trained NN depth estimation module to compute a second category depth estimate or z coordinate for selected x, y coordinates in said first and second plurality of images, wherein said second category depth computation module outputs a 3D model of structures from within said first and second plurality of images.

An autonomous mobile system comprising:

• a vehicle comprising an engine, navigation system, a machine vision system, and vehicle maneuver control elements that are controlled based on inputs from the machine vision system and navigation system, wherein the machine vision system comprises a system for executing a plurality of machine instructions that execute a method for producing a three dimensional model data from multi-perspective two dimensional images, the method comprising:
• capturing a first and second plurality of images from a first and second camera that are oriented with a first and second overlapping fields of view (FOV) with a first controller section;
• determining or setting camera settings comprising focus, exposure, and zoom with a second controller section;
• identifying a plurality of features within two or more sets of the first and second plurality of images using a third control section comprising a feature detection system to identify point correspondences (PCs) in the two or more sets of the first and second plurality of images and store a first and second list of x, y coordinates of PCs associated with each feature found in at least one corresponding pair of said first and second plurality of images;
• performing a first and second depth estimation processing using a fourth controller section respectively comprising performing structure from motion (SFM) processing and depth from defocus (DFD) processing to respectively produce a plurality of depth estimates for each PC and a relative blur estimate around each PC, wherein said DFD processing comprises determining relative blur estimates around said PCs for pairs of points within each of said sets of the first and second plurality of images;
• defining or using a provided first polynomial algorithm (PA) defining or describing a projective camera model using a fifth controller section including respective relationships between x, y coordinates within sets of PCs and said first and second cameras and outputting first and second ratios of depth (RoDs), a list of coefficients for variables in the projective camera model that describe relationships between three dimensional coordinates of arbitrary sets or pairs of combinations of PCs and the first PA;
• defining or using a provided second PA using a sixth controller section describing relationships between PCs and said first and second cameras based on a thin lens equation and computing relative blur ratios using sets of RBEs divided by each other that is used to cancel an element of the second PA;
• executing a simplification processing machine instruction sequence using a seventh controller section to perform variable elimination or reduction of a system of equations (SoE) comprising said first and second PAs comprising a data entry operation comprising substitution of said first and second RoDs into said first PA and second PA to eliminate a first and second variable and output a first simplified SoE (FSSoE) multivariate polynomial;
• solving said FSSoE using an eighth controller section by finding roots of the FSSoE using a system of Sylvester Resultant univariate polynomials to eliminate another variable and generating a second simplified SOE (SSSoE) and multiple possible solutions to the SSSoE;
• determining a best solution for each of said PCs using a ninth controller section by back-substituting each of said possible solutions into each Sylvester Resultant univariate polynomial or the SSSoE and calculating a mean result of the back substitution while excluding outliers of the back substitution to obtain an algebraic or geometric (A/G) error estimate of each depth estimate to produce a first category depth estimate;
• computing and outputting a three-dimensional (3D) model of image structures within said first and second camera overlapping FOVs, wherein said 3D model comprises a second category depth estimate for selected x,y pixels in input two-dimensional (2D) images;
• and
• operating a reinforcement learning camera settings selection module comprising a state extraction module, a cost measurement module, and an action selection module for selecting or determining said camera settings;
• and
• operating a second category depth computation module comprising a monocular depth estimation system comprising a depth estimation neural network (NN) training module and a trained real time depth estimation module which uses a trained NN depth estimation module output from said depth estimation NN training module, wherein said depth estimation NN training module comprises a first deep convolutional or auto-encoder feature extraction layers with patterned input sampling training module and a first depth prediction layer training module configured to compute and output said trained NN depth estimation module, wherein said trained real time depth estimation module comprises a deep convolutional or auto-encoder feature extraction layer with patterned input sampling and a second depth prediction layer module configured to use said trained NN depth estimation module to compute a second category depth estimate or z coordinate for selected x, y coordinates in said first and second plurality of images, wherein said second category depth computation module outputs a 3D model of structures from within said first and second plurality of images.

A machine vision system comprising:

• a first and second camera system respectively configured to have a first and second overlapping field of view (FOV);
• a first control module comprising a camera settings and image capture module that outputs a plurality of first and second image sets and a camera settings data structure;
• a second control module comprising an image feature detector and matching module configured to identify image features and associated x, y coordinates from said first and second image sets based on a predetermined one or more image features;
• a third control module comprising a first category depth estimator module selector or bypass system;
• a fourth control module comprising an image calibration module configured to optionally perform image calibration operations on said plurality of first and second image sets;
• a fifth control module comprising a first category depth computation module configured to compute and output a neural network (NN) training input dataset comprising said first and second image sets and associated three-dimensional (3D) x, y, and first category z coordinates associated with said identified image features;
• a sixth control module comprising a second category depth computation module configured to compute and output a 3D model of image structures within said first and second camera overlapping FOVs including a second category depth estimate for selected x,y pixels in input two-dimensional (2D) images;
• and
• a seventh control module comprising a reinforcement learning camera settings selection module;
• wherein said first category depth computation module comprises a structure from motion (SFM) computation module, a first polynomial algorithm (PA) computation module comprising a projective camera model based PA, a depth from defocus (DFD) module comprising a blur estimation module and a second PA computation module, a first variable elimination module (VEM) configured to eliminate a first and second depth variable from first and second PAs, a second VEM including a Sylvester Resultant Module for eliminating a third depth variable from the first and second PAs, and an error and depth computation module for determining algebraic or geometric error (A/G);
• wherein said second category depth computation module comprises a monocular depth estimation system comprising a depth estimation NN training module and a trained real time depth estimation module which uses a trained NN depth estimation module output from said depth estimation NN training module;
• wherein said depth estimation NN training module comprises a first deep convolutional or auto-encoder feature extraction layers with patterned input sampling training module and a first depth prediction layer training module configured to compute and output said trained NN depth estimation module;
• wherein said trained real time depth estimation module comprises a deep convolutional or auto-encoder feature extraction layer with patterned input sampling and a second depth prediction layer module configured to use said trained NN depth estimation module to compute a second category depth estimate or z coordinate for selected x, y coordinates in said first and second image sets;
• wherein said reinforcement learning camera settings selection module comprises a state extraction module, a cost measurement module, and an action selection module for selecting or determining said camera settings.

An autonomous mobile system comprising:

• a vehicle comprising an engine, navigation system, a machine vision system, and vehicle maneuver control elements that are controlled based on inputs from the machine vision system and navigation system, wherein the machine vision system comprises:
• a first and second camera system respectively configured to have a first and second overlapping field of view (FOV);
• a first control module comprising a camera settings and image capture module that outputs a plurality of first and second image sets and a camera settings data structure;
• a second control module comprising an image feature detector and matching module configured to identify image features and associated x, y coordinates from said first and second image sets based on a predetermined one or more image features;
• a third control module comprising a first category depth estimator module selector or bypass system;
• a fourth control module comprising an image calibration module configured to optionally perform image calibration operations on said plurality of first and second image sets;
• a fifth control module comprising a first category depth computation module configured to compute and output a neural network (NN) training input dataset comprising said first and seconds image sets and associated three-dimensional (3D) x, y, and first category z coordinates associated with said identified image features;
• a sixth control module comprising a second category depth computation module configured to compute and output a 3D model of image structures within said first and second camera overlapping FOVs including a second category depth estimate for selected x,y pixels in input two-dimensional (2D) images;
• and
• a seventh control module comprising a reinforcement learning camera settings selection module;
• wherein said first category depth computation module comprises a structure from motion (SFM) computation module, a first polynomial algorithm (PA) computation module comprising a projective camera model based PA, a depth from defocus (DFD) module comprising a blur estimation module and a second PA computation module, a first variable elimination module (VEM) configured to eliminate a first and second depth variable from first and second PAs, a second VEM including a Sylvester Resultant Module for eliminating a third depth variable from the first and second PAs, and an error and depth computation module for determining algebraic or geometric error (A/G);
• wherein said second category depth computation module comprises a monocular depth estimation system comprising a depth estimation NN training module and a trained real time depth estimation module which uses a trained NN depth estimation module output from said depth estimation NN training module;
• wherein said depth estimation NN training module comprises a first deep convolutional or auto-encoder feature extraction layers with patterned input sampling training module and a first depth prediction layer training module configured to compute and output said trained NN depth estimation module;
• wherein said trained real time depth estimation module comprises a deep convolutional or auto-encoder feature extraction layer with patterned input sampling and a second depth prediction layer module configured to use said trained NN depth estimation module to compute a second category depth estimate or z coordinate for selected x, y coordinates in said first and second image sets;
• wherein said reinforcement learning camera settings selection module comprises a state extraction module, a cost measurement module, and an action selection module for selecting or determining said camera settings.