Performing object modeling by combining visual data from images with motion data of the image acquisition device

A computer-implemented method comprising:

• obtaining, by one or more computing devices, information about a succession of images captured at multiple positions around an exterior of a pile of material that is deposited at a fixed site and about metadata from capturing of the succession of images, wherein the metadata is based at least in part on data from one or more motion sensors of at least one device involved in the capturing of the succession of images and includes a plurality of estimated pose data points with an estimated pose data point for each image of the succession indicating location and orientation of the at least one device during the capturing of that image;
• generating, by the one or more computing devices, and based on a combination of the metadata and of visual data from at least some images of the succession, a three-dimensional (“3D”) computer model representing at least some of the exterior of the pile of material, including:
• determining, by the one or more computing devices and based at least in part on estimated pose data points for images of the succession, initial trajectory information that indicates at least one sequence of estimated pose data points of the at least one device during the capturing of some of the images of the succession, including excluding from the initial trajectory information one or more estimated pose data points that do not satisfy one or more defined trajectory criteria;
• determining, by the one or more computing devices and based at least in part on analysis of visual data from images of the succession, an image sequence having multiple images that are a subset of the images of the succession and such that each image in the image sequence has, for one or more adjacent images in the image sequence, an amount of visual overlap with the one or more adjacent images that satisfies one or more defined visual overlap criteria;
• determining, by the one or more computing devices, updated trajectory information for the multiple images of the image sequence that includes pose data points with revised location and orientation values, including modifying the estimated pose data points of the initial trajectory information based on triangulations across images of the image sequence of positions of features that are each visible in at least two images of the image sequence, and including further optimizing the modified estimated pose data points based on one or more specified pose constraints;
• determining, by the one or more computing devices and based at least in part on the updated trajectory information, positions of a plurality of 3D points on the exterior of the pile of material, including, for each of multiple 3D points that are visible in two or more images of the image sequence, adjusting an initial alignment of that 3D point's position across the two or more images to optimize an overall alignment of visual data of the multiple images of the image sequence;
• and
• using, by the one or more computing devices, the determined positions of the plurality of 3D points to generate the 3D computer model representing the at least some of the exterior of the pile of material at a time of the capturing;
• and
• providing, by the one or more computing devices, information about the generated 3D computer model.

A non-transitory computer-readable medium having stored contents that cause one or more computing devices to perform automated operations including at least:

• obtaining, by the one or more computing devices, information about a plurality of images captured during an image acquisition session at multiple positions around an exterior of a pile of material that is deposited at a fixed site and about metadata from capturing of the plurality of images, wherein the one or more computing devices include a mobile computing device that includes one or more motion sensors and includes one or more imaging sensors and is used for the capturing of the plurality of images, wherein the metadata is based at least in part on data acquired from the one or more motion sensors and includes an estimated pose data point for each image of the plurality that indicates location and orientation of the mobile computing device during the capturing of that image;
• generating, by the one or more computing devices and during the image acquisition session before additional images are captured for the image acquisition session at multiple additional positions around the exterior of the pile of material, and based on a combination of the metadata and of visual data from at least some images of the plurality, a partial three-dimensional (“3D”) computer model representing some of the exterior of the pile of material, including:
• determining, by the one or more computing devices and using estimated pose data points for images of the plurality, initial trajectory information that indicates at least one sequence of estimated pose data points of the mobile computing device during the capturing of some images of the plurality;
• determining, by the one or more computing devices and based at least in part on analysis of visual data from images of the plurality, an image sequence having multiple images that are a subset of the plurality of images and such that each image in the image sequence has, for one or more adjacent images in the image sequence, an amount of visual overlap with the one or more adjacent images that satisfies one or more defined visual overlap criteria;
• determining, by the one or more computing devices, updated trajectory information for the multiple images of the image sequence that includes pose data points with revised location and orientation values, including modifying the estimated pose data points of the initial trajectory information based on triangulations across images of the image sequence of positions of features that are each visible in at least two images of the image sequence;
• determining, by the one or more computing devices and based at least in part on the updated trajectory information, positions of a plurality of 3D points on the exterior of the pile of material, including, for each of multiple 3D points that are visible in two or more images of the image sequence, aligning that 3D point's position across the two or more images;
• and
• using, by the one or more computing devices, the determined positions of the plurality of 3D points to generate the partial 3D computer model representing some of the exterior of the pile of material;
• and
• displaying, on the mobile computing device, and during the capturing of the additional images to a user participating in the capturing of the additional images, at least some of the partial generated 3D computer model overlaid on a current image of the pile of material, wherein the displayed at least some partial generated 3D computer model includes at least one of a 3D surface corresponding to the at least some of the generated 3D computer model or some or all of the plurality of 3D points.

A system comprising:

• one or more hardware processors of one or more computing devices;
• and
• one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing devices to perform automated operations including at least:
• obtaining information about a plurality of images captured during an image acquisition session at multiple positions around an exterior of a pile of material that is deposited at a fixed site and about metadata from capturing of the plurality of images, wherein the metadata is based at least in part on data from one or more motion sensors of at least one device involved in the capturing of the succession of images and includes a plurality of estimated pose data points with an estimated pose data point for each image of the succession indicating location and orientation of the at least one device during the capturing of that image;
• generating, based on a combination of the metadata and of visual data from at least some images of the plurality, a completed three-dimensional (“3D”) computer model representing the exterior of the pile of material, including:
• determining, using estimated pose data points for images of the plurality, initial trajectory information that indicates at least one sequence of estimated pose data points of the at least one device during the capturing of at least some images of the plurality;
• determining, based at least in part on analysis of visual data from images of the plurality, an image sequence having multiple images that are a subset of the plurality of images and such that each image in the image sequence has, for one or more adjacent images in the image sequence, an amount of visual overlap with the one or more adjacent images that satisfies one or more defined visual overlap criteria;
• determining updated trajectory information for the multiple images of the image sequence that includes pose data points with revised location and orientation values, including modifying the estimated pose data points of the initial trajectory information based on triangulations across images of the image sequence of positions of features that are each visible in at least two images of the image sequence;
• determining, based at least in part on the updated trajectory information, positions of a plurality of 3D points on the exterior of the pile of material, including, for each of multiple 3D points that are visible in two or more images of the image sequence, aligning that 3D point's position across the two or more images;
• and
• using the determined positions of the plurality of 3D points to generate the completed 3D computer model representing the exterior of the pile of material;
• and
• transmitting, to one or more client devices, at least some of the completed generated 3D computer model, to enable each of the one or more client devices to display the at least some completed generated 3D computer model overlaid on a current image of an environment of that client device, wherein the displayed at least some completed generated 3D computer model includes at least one of a 3D surface corresponding to the at least some of the completed generated 3D computer model or some or all of the plurality of 3D points.