US 11,443,516 B1 · Locally and globally locating actors by digital cameras and machine learning

Abstract

Motion of actors within a scene may be detected based on imaging data, using machine learning tools operating on cameras that captured the imaging data. The machine learning tools process images to perform a number of tasks, including detecting heads of actors, and sets of pixels corresponding to the actors, before constructing line segments from the heads of the actors to floor surfaces on which the actors stand or walk. The line segments are aligned along lines extending from locations of heads within an image to a vanishing point of a camera that captured the image. Trajectories of actors and visual data are transferred from the cameras to a central server, which links trajectories captured by multiple cameras and locates detected actors throughout the scene, even when the actors are not detected within a field of view of at least one camera.

A system comprising:

a first camera including at least a first portion of a scene within a first field of view, wherein the first camera comprises a first processor unit and a first optical sensor;
a second camera including at least a second portion of the scene within a second field of view, wherein the second camera comprises a second processor unit and a second optical sensor;
and
a computer system in communication with the first camera and the second camera,
wherein the computer system is programmed with one or more sets of instructions that, when executed by the computer system, cause the computer system to at least:
receive a first camera trajectory for a first object comprising a first set of positions of the first object and a first object identifier from the first camera, wherein the first camera trajectory is generated by the first camera based at least in part on:
a location of a first set of pixels representing the first object depicted within a first image captured by the first camera at a first time;
and
a location of a second set of pixels representing the first object depicted within a second image captured by the first camera at a second time, wherein a second set of visual descriptors of the second set of pixels is similar to a first set of visual descriptors of the first set of pixels,
wherein a first one of the first set of positions corresponds to a first point on a ground plane determined based at least in part on a first line segment extending from a location of a head depicted within the first set of pixels to a location corresponding to the first point on the ground plane;
receive a second camera trajectory for a second object comprising a second set of positions of the second object and a second object identifier from the second camera, wherein the second camera trajectory is generated by the second camera based at least in part on:
a location of a third set of pixels representing the second object depicted within a third image captured by the second camera at the first time;
and
a location of a fourth set of pixels representing the second object depicted within a fourth image captured by the second camera at the second time, wherein a fourth set of visual descriptors of the fourth set of pixels is similar to a third set of visual descriptors of the third set of pixels,
wherein a second one of the second set of positions corresponds to a second point on the ground plane determined based at least in part on a second line segment extending from a location of a head depicted within the second set of pixels to a location corresponding to the second point on the ground plane;
determine that the first one of the first set of positions is consistent with the second one of the second set of positions;
and
in response to determining that the first one of the first set of positions is consistent with the second one of the second set of positions,
generate a first system trajectory based at least in part on the first camera trajectory and the second camera trajectory;
and
store the first system trajectory in association with the first object identifier and the second object identifier in at least one data store.

18 more elements

Locally and globally locating actors by digital cameras and machine learning

Life of this patent

Locally and globally locating actors by digital cameras and machine learning

Life of this patent

Abstract

Claims20