# US 2021/0279659 A1 · Construction site productivity capture using computer vision

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279659 A1
- Document ID: US20210279659A1
- Status: Published application
- Kind: A1
- Application number: 16/813,609
- Filed: 2020-03-09
- Published: 2021-09-09
- Earliest claimed: 2020-03-09
- Assignees: Trimble Inc.
- Applicants: Trimble Inc.
- Inventors: Karoliina Torttila; Jim Coleman; Robert Banfield; Matt Compton
- Web page: https://patentgrep.com/patent/US20210279659A1

## CPC

- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- E02F 3/435: {for dipper-arms, backhoes or the like}
- E02F 9/205: {Remotely operated machines, e.g. unmanned vehicles (E02F3/8866 takes precedence)}
- E02F 9/2054: {Fleet management}
- E02F 9/262: {with follow-up actions to control the work tool, e.g. controller}
- G06Q 10/0639: Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q 50/165: {Land development}
- G06T 7/13: Edge detection
- G06T 7/74: {involving reference images or patches}
- G06F 18/24133: {Distances to prototypes}
- G06T 2207/30164: Workpiece; Machine component
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- E02F: DREDGING; SOIL-SHIFTING (winning peat E21C49/00)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments describe a method for capturing objects in action at an earthmoving site. The method includes capturing an image of a region of the earthmoving site including an object, the image being recorded as captured digital data; identifying a classification of the object using a trained algorithm existing in memory of the image capturing device; sending the classification of the object to a remote server through a network; determining a pixel location and a boundary of the object within a field of view of the image capturing device based on positions of pixels of the object in the image; sending a set of images including the image to the remote server; determining an activity being performed by the object based on an analysis of digital data associated with the classification of the object and the movement of the object at the earthmoving site; and outputting a report to a user.

## Claims

### Claim 1

A method for capturing objects in action at an earthmoving site, the method comprising:
capturing, by an image capturing device positioned at the earthmoving site, an image of a region of the earthmoving site including an object, the image being recorded as captured digital data;
identifying a classification of the object using a trained algorithm existing in memory of the image capturing device;
sending, by the image capturing device, the classification of the object to a remote server through a network;
determining a pixel location and a boundary of the object within a field of view of the image capturing device based on positions of pixels of the object in the image;
sending, by the image capturing device, a set of images including the image to the remote server, the set of images representing movement of the object at the earthmoving site;
determining an activity being performed by the object based on an analysis of digital data associated with the classification of the object and the movement of the object at the earthmoving site; and
outputting a report to a user based on the activity performed by the object at the earthmoving site.

### Claim 2 (depends on 1)

The method of claim 1, wherein the object is a first object, the pixel location is a first pixel location, the boundary is a first boundary, the image capturing device is a first image capturing device, and the image includes the first object and a second obj ect.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
determining a second pixel location and a second boundary of the second object within the field of view of the image capturing device based on positions of pixels of the second object in the image;
determining a first three-dimensional (3D) position of the first object in the earthmoving site based on the first pixel location in the field of view of the first image capturing device and a third pixel location in a field of view of a second image capturing device;
determining a second 3D position of the second object in the earthmoving site based on the second pixel location in the field of view of the first image capturing device and a fourth pixel location in the field of view of the second image capturing device;
determining a distance between the first 3D position and the second 3D position;
comparing the distance to a proximity threshold indicating a minimum distance between the first and second objects; and
in response to a determination that the distance is below the proximity threshold, performing the sending of the set of images to the remote server.

### Claim 4 (depends on 3)

The method of claim 3, wherein the set of images include the first object and the second object when the distance between the first 3D position and the second 3D position are below the proximity threshold.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
generating, by the image capturing device, a warning signal in response to the determination that the distance is below the proximity threshold.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining that the classification of the object is one of a set of predefined objects; and
in response to the determination that the classification of the object is the one of the set of predefined objects, performing the sending of the set of images to the remote server.

### Claim 7 (depends on 1)

The method of claim 1, wherein the image capturing device is a first image capturing device positioned at a first physical location within the earthmoving site, the image is a first image, the region is a first region of the earthmoving site, the captured digital data is a first captured digital data, and wherein the method further includes:
capturing, by a second image capturing device positioned at a second physical location within the earthmoving site different from the first physical location, a second image of a second region of the earthmoving site including the object from a different perspective than the first image capturing device, the second image being recorded as second captured digital data; and
using a first known position of the first image capturing device and second known position of the second image capturing device, along with a known 3D position of a common point captured within fields of view of the first and second image capturing devices, to determine a position of the object at the earthmoving site in 3D space.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining the pixel location and the boundary of the object is performed by the image capturing device at the earthmoving site.

### Claim 9 (depends on 8)

The method of claim 8, further comprising sending, by the image capturing device, the pixel location and the boundary of the object into the remote server through the network.

### Claim 10 (depends on 1)

The method of claim 1, wherein the boundary is a boundary box that forms a rectangle around the object and that dynamically moves in location and dimension corresponding to a movement of the object through the set of images.

### Claim 11 (depends on 1)

The method of claim 1, wherein the object is at least one of a group consisting of heavy equipment and personnel.

### Claim 12

An image capturing device for capturing objects in action at an earthmoving site, the device comprising:
a camera for capturing images of a region of the earthmoving site;
communication circuitry operable to send data to a remote server;
memory for storing at least a portion of the captured images of the earthmoving site, the memory comprising an image classifier module or an object detection module; and
a processor coupled to the memory, the camera, and the communication circuitry, the processor capable of executing instructions in the memory to:
capture an image of the region of the earthmoving site including an object, the image being recorded as captured digital data;
identify a classification of the object using a trained algorithm of the image classifier module or the object detection module existing in the memory;
send the classification of the object to the remote server through a network;
determine, by the object detection module, a pixel location and a boundary of the object within a field of view of the image capturing device based on positions of pixels of the object in the image;
send a set of images including the image to the remote server, the set of images representing a movement of the obj ect at the earthmoving site;
determine an activity being performed by the object based on an analysis of digital data associated with the classification of the object and the movement of the object at the earthmoving site; and
output a report to a user based on the activity performed by the object at the earthmoving site.

### Claim 13 (depends on 12)

The image capturing device of claim 12, wherein the object is a first object, the pixel location is a first pixel location, the boundary is a first boundary, the image capturing device is a first image capturing device, and the image includes the first object and a second object.

### Claim 14 (depends on 13)

The image capturing device of claim 13, wherein the processor is further configured to:
determine a second pixel location and a second boundary of the second object within the field of view of the image capturing device based on positions of pixels of the second object in the image;
determine a first three-dimensional (3D) position of the first object in the earthmoving site based on the first pixel location in the field of view of the first image capturing device and a third pixel location in a field of view of a second image capturing device;
determine a second 3D position of the second object in the earthmoving site based on the second pixel location in the field of view of the first image capturing device and a fourth pixel location in the field of view of the second image capturing device;
determine a distance between the first 3D position and the second 3D position;
compare the distance to a proximity threshold indicating a minimum distance between the first and second objects; and
in response to a determination that the distance is below the proximity threshold, sending the set of images to the remote server.

### Claim 15 (depends on 14)

The image capturing device of claim 14, wherein the image includes the first object and the second object when the distance between the first 3D position and the second 3D position are below the proximity threshold.

### Claim 16 (depends on 14)

The image capturing device of claim 14, wherein the processor is further configured to:
generate a warning signal in response to the determination that the distance is below the proximity threshold.

### Claim 17

A system for capturing an object in action at an earthmoving site, the system comprising:
a first image capturing device positioned within the earthmoving site and operable to capture first images of a first region of the earthmoving site; and
a second image capturing device positioned within the earthmoving site and configured to capture second images of a second region of the earthmoving site, wherein the first region overlaps with at least a portion of the second region, and wherein the first image capturing device and the second image capturing device are communicatively coupled to a remote server over a network,
wherein the first image capturing device comprises:
a camera for capturing the first images of the first region;
communication circuitry operable to send data to the remote server;
memory for storing at least a portion of the captured first images of the earthmoving site, the memory comprising an image classifier module and an object detection module; and
a processor coupled to the memory, the camera, and the communication circuitry, the processor capable of executing instructions in the memory to:
capture an image of a region of the earthmoving site including an object, the image being recorded as captured digital data;
identify a classification of the object using a trained algorithm existing in the memory of the first image capturing device;
send the classification of the object to the remote server through the network;
determine a pixel location and a boundary of the object within a field of view of the first image capturing device based on positions of pixels of the object in the image;
send a set of images including the image to the remote server, the set of images representing a movement of the object at the earthmoving site; and
determine an activity being performed by the object based on an analysis of digital data associated with the classification of the object and the movement of the object at the earthmoving site,
wherein a report based on the activity performed by the object at the earthmoving site is outputted to a user.

### Claim 18 (depends on 17)

The system of claim 17, wherein the first image capturing device is communicatively coupled to the second image capturing device so that data can be transferred between the first and second image capturing devices.

### Claim 19 (depends on 17)

The system of claim 17, wherein the object is a first object, the pixel location is a first pixel location, the boundary is a first boundary, and the image includes the first object and a second object.

### Claim 20 (depends on 19)

The system of claim 19, wherein the processor is further configured to:
determine a second pixel location and a second boundary of the second object within the field of view of the first image capturing device based on positions of pixels of the second object in the image;
determine a first three-dimensional (3D) position of the first object in the earthmoving site based on the first pixel location in the field of view of the first image capturing device and a third pixel location in a field of view of the second image capturing device;
determine a second 3D position of the second object in the earthmoving site based on the second pixel location in the field of view of the first image capturing device and a fourth pixel location in the field of view of the second image capturing device;
determine a distance between the first 3D position and the second 3D position;
compare the distance to a proximity threshold indicating a minimum distance between the first and second objects; and
in response to a determination that the distance is below the proximity threshold, sending the set of images to the remote server.
