US 12,743,795 B2Granted

Determining object structure using camera devices with views of moving objects

AssigneeEveryPoint, Inc.

InventorsDavid B. Boardman, Jared S. Heinly, Srinivas Kapaganty, Brooke N. Steele

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques are described for automated analysis and use of data acquired about a moving object of interest, such as from one or more physically mounted cameras that have at least partial coverage of the object exterior, to automatically generate a computer model of the object from visual data in images and to use the computer model to automatically estimate values for one or more object attributes. For example, the described techniques may include, for a pile of material being moved past one or more stationary cameras by one or more transporting vehicles, using acquired images that provide visual coverage of only a subset of the pile's exterior to measure the volume of the pile. The images from such devices may be acquired at various times (e.g., when triggered by object movement), and may be used to monitor movement trajectories and other attributes of one or more such objects.

Claims21

Dependent claims

A computer-implemented method comprising:

acquiring, during a period of time and from a stationary camera that is physically mounted at a fixed location and a fixed orientation, and as a transporting vehicle moves a pile of material through multiple successive locations by the stationary camera during the period of time, multiple images of an exterior of the pile of material at the multiple locations that in aggregate have visual data for the exterior of the pile of material;
generating, by one or more computing devices and using Structure-from-Motion (SfM) processing, a three-dimensional (“3D”) computer model to represent the exterior of the pile of material, including:
analyzing, by the one or more computing devices, the visual data of the multiple images to determine multiple respective shapes of the pile of material, including determining, for each of the multiple images, a shape of a portion of the exterior of the pile of material that is visible in that image;
tracking, by the one or more computing devices, features of the pile of material at different positions in the multiple images over the period of time;
aligning, by the one or more computing devices and based at least in part on the tracked features, the multiple images into a common coordinate system;
and
combining, by the one or more computing devices and using the aligned multiple images, the determined multiple shapes of the pile of material to produce the 3D computer model;
and
determining, by the one or more computing devices, a trajectory of movement of the pile of material as it moves through the multiple successive locations, including:
obtaining, by the one or more computing devices and from the SfM processing, a plurality of respective distinct estimated camera poses for the stationary camera calculated for the multiple images;
and
using, by the one or more computing devices, the plurality of respective distinct estimated camera poses for the stationary camera to determine the trajectory of the pile of material;
determining, by the one or more computing devices and based at least in part on the 3D computer model, one or more attributes of the pile of material that include at least a volume of the pile of material;
and
providing, by the one or more computing devices, information about the pile of material for use on one or more client devices that includes the determined one or more attributes and further includes the determined trajectory.

12 more elements

A computer-implemented method comprising:

acquiring, during a period of time and from one or more cameras each physically mounted at a respective fixed location, and as an object is successively at multiple locations as the object moves by the one or more cameras during the period of time, multiple images of an exterior of the object at the multiple locations that in aggregate have visual data for some or all of the exterior of the object;
generating, by one or more computing devices, a three-dimensional (“3D”) computer model to represent at least some of the exterior of the object, including:
analyzing, by the one or more computing devices, the visual data of the multiple images to determine multiple respective shapes of the object, including determining, for each of the multiple images, a shape of a portion of the exterior of the object that is visible in that image;
tracking, by the one or more computing devices, features of the object at different positions in the multiple images over the period of time;
aligning, by the one or more computing devices and based at least in part on the tracked features, the multiple images into a common coordinate system;
and
combining, by the one or more computing devices and using the aligned multiple images, the determined multiple shapes of the object to produce the 3D computer model;
determining, by the one or more computing devices, a trajectory of movement of the object for the multiple locations, including:
determining, by the one or more computing devices, a plurality of respective distinct estimated camera poses for the one or more cameras calculated for the multiple images;
and
using, by the one or more computing devices, the plurality of respective distinct estimated camera poses for the one or more cameras to determine the trajectory of the object;
and
providing, by the one or more computing devices, information about the 3D computer model and about the determined trajectory for use on one or more client devices.

10 more elements

A non-transitory computer-readable medium having stored contents that cause one or more computing devices to perform automated operations including at least:

acquiring, during a period of time and from one or more cameras, and as an object is successively at multiple locations as the object moves by the one or more cameras during the period of time, multiple images of an exterior of the object at the multiple locations that in aggregate have visual data for some or all of the exterior of the object;
generating, by the one or more computing devices, a three-dimensional (“3D”) computer model to represent at least some of the exterior of the object, including:
analyzing, by the one or more computing devices, the visual data of the multiple images to determine multiple respective shapes of the object, including determining, for each of the multiple images, a subset of pixels of that image that correspond to the object, and a shape of a portion of the exterior of the object that is visible in the subset of pixels of that image;
tracking, by the one or more computing devices and using the determined subset of pixels for each of the multiple images, features of the object at different positions in the multiple images over the period of time;
aligning, by the one or more computing devices and based at least in part on the tracked features, the multiple images into a common coordinate system;
and
combining, by the one or more computing devices and using the aligned multiple images, the determined multiple shapes of the object to produce the 3D computer model;
and
determining, by the one or more computing devices, a trajectory of the object for the multiple locations, including:
determining, by the one or more computing devices, a plurality of respective distinct estimated camera poses for the one or more cameras calculated for the multiple images;
and
using, by the one or more computing devices, the plurality of respective distinct estimated camera poses for the one or more cameras to determine the trajectory of the object;
and
providing, by the one or more computing devices, information about the 3D computer model and the determined trajectory for use on one or more client devices.

11 more elements

A system comprising:

one or more hardware processors of one or more computing devices;
and
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing devices to perform automated operations including at least:
obtaining multiple images acquired of an object during a period of time, wherein the multiple images are acquired from one or more cameras each physically mounted at a respective fixed location as the object is successively at multiple locations during the period of time, and wherein the multiple images in aggregate include visual data of some or all of an exterior of the object at the multiple locations;
generating a three-dimensional (“3D”) computer model to represent at least some of the exterior of the object, including:
analyzing the visual data of the multiple images to determine multiple respective shapes of the object, including determining, for each of the multiple images, a shape of a portion of the exterior of the object that is visible in that image;
matching features of the object at different positions in the multiple images over the period of time;
aligning, based at least in part on the matched features, the multiple images into a common coordinate system;
and
combining, using the aligned multiple images, the determined multiple shapes of the object to produce the 3D computer model;
determining a trajectory of the object for the multiple locations, including:
determining a plurality of respective distinct estimated camera poses for the one or more cameras calculated for the multiple images;
and
using the plurality of respective distinct estimated camera poses for the one or more cameras to determine the trajectory of the object;
and
providing information about the 3D computer model including the determined trajectory.

13 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/661,612Filed 11 May 2024 · continuation in part
Priority: 9 Feb 2021
Published: US 2024/0303833 A112 Sep 2024
Granted: 22 Sep 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification20 codes

G06T 7/251
Using feature-based methods, e.g. the tracking of corners or segments: involving models
G06T 7/55
Depth or shape recovery: from multiple images
G06T 7/344
Using feature-based methods: involving models
G06T 7/579
From multiple images: from motion

Family

Application17/171,808Parent of this continuation-in-part · filed 9 Feb 2021
Application17/591,515Parent of this continuation-in-part · filed 2 Feb 2022
US 11,282,291 B1Parent of this continuation-in-part · filed 2 Mar 2022
US 2024/0303833 A1Pre-grant publication · published 12 Sep 2024
US 11,989,899 B2Parent of this continuation-in-part
US 2022/0254045 A1Published application
US 12,743,795 B2This patent

From the US filing history. Foreign family members aren’t listed.