# US 2021/0174548 A1 · Calibrating cameras using human skeleton

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174548 A1
- Document ID: US20210174548A1
- Status: Published application
- Kind: A1
- Application number: 17/249,049
- Filed: 2021-02-18
- Published: 2021-06-10
- Earliest claimed: 2019-01-29
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Hongli DENG; Zicheng LIU
- Web page: https://patentgrep.com/patent/US20210174548A1

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 7/292: Multi-camera tracking
- G06T 7/579: from motion
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06T 2207/20044: Skeletonization; Medial axis transform
- G06T 2207/20072: Graph-based image processing
- G06T 2207/20164: Salient point detection; Corner detection
- G06T 2207/30196: Human being; Person
- G06T 2207/30232: Surveillance
- G06T 2207/30241: Trajectory
- G06T 2207/30244: Camera pose
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Examples are disclosed herein that relate to automatically calibrating cameras based on human detection. One example provides a computing system comprising instructions executable to receive image data comprising depth image data and two-dimensional image data of a space from a camera, detect a person in the space via the image data, determine a skeletal representation for the person via the image data, determine over a period of time a plurality of locations at which a reference point of the skeletal representation is on a ground area in the image data, determine a ground plane of the three-dimensional representation based upon the plurality of locations at which the reference point of the skeletal representation is on the ground area in the image data, and track a location of an object within the space relative to the ground plane.

## Claims

### Claim 1

A computing system comprising:
a processor; and
storage comprising instructions executable by the processor to:
receive image data comprising depth image data and two-dimensional image data of a space from a camera,
detect a person in the space via the image data,
determine a skeletal representation for the person via the image data,
determine over a period of time a plurality of locations at which a reference point of the skeletal representation is on a ground area in the image data,
determine a ground plane of the three-dimensional representation based upon the plurality of locations at which the reference point of the skeletal representation is on the ground area in the image data, and
track a location of an object within the space relative to the ground plane.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein the reference point of the skeletal representation is based on a lowest node of the skeletal representation.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein the instructions are further executable to define a convex hull that includes the plurality of locations at which the reference point of the skeletal representation is on the ground area in the image data, map a plurality of points within the convex hull to a three-dimensional representation of the space to form a point cloud, and determine the ground plane based upon the point cloud.

### Claim 4 (depends on 3)

The computing system of claim 3, wherein the instructions are further executable to form the point cloud by mapping a plurality of two-dimensional points within the convex hull as determined from the two-dimensional image data to the three-dimensional representation of the space.

### Claim 5 (depends on 1)

The computing system of claim 1, wherein the instructions are executable to receive image data from a plurality of cameras and to determine the ground plane based upon the image data from the plurality of cameras.

### Claim 6 (depends on 5)

The computing system of claim 5, wherein the instructions are executable to spatially calibrate a pair of cameras of the plurality of cameras based at least upon a skeleton position of the person in an image from a first camera of the pair of cameras and the skeleton position of the person in an image from a second camera of the pair of cameras acquired when the person is located in the overlapping field of view.

### Claim 7 (depends on 6)

The computing system of claim 6, wherein the instructions are executable to spatially calibrate the pair of cameras further based upon the ground plane.

### Claim 8

A method of tracking an object, the method comprising:
receiving image data comprising depth image data and two-dimensional image data of a space from a camera;
detecting a person in the space via the image data;
determining a skeletal representation for the person via the image data;
determining over a period of time a plurality of locations at which a reference point of the skeletal representation is on a ground area in the image data;
determining a ground plane of the three-dimensional representation based upon the plurality of locations at which the reference point of the skeletal representation is on the ground area in the image data; and
tracking a location of an object within the space relative to the ground plane.

### Claim 9 (depends on 8)

The method of claim 8, further comprising defining a convex hull that includes the plurality of locations at which the reference point of the skeletal representation is on the ground area in the image data, mapping a plurality of points within the convex hull to a three-dimensional representation of the space to form a point cloud, and determining the ground plane based upon the point cloud.

### Claim 10 (depends on 8)

The method of claim 8, further comprising forming the point cloud by mapping a plurality of two-dimensional points within the convex hull as determined from the two-dimensional image data to the three-dimensional representation of the space.

### Claim 11 (depends on 8)

The method of claim 8, further comprising receiving image data from a plurality of cameras and to determine the ground plane based upon the image data from the plurality of cameras.

### Claim 12 (depends on 8)

The method of claim 8, further comprising spatially calibrating a pair of cameras of the plurality of cameras based at least upon a skeleton position of the person in an image from a first camera of the pair of cameras and the skeleton position of the person in an image from a second camera of the pair of cameras acquired when the person is located in the overlapping field of view.

### Claim 13 (depends on 8)

The method of claim 8, further comprising spatially calibrating the pair of cameras further based upon the ground plane.

### Claim 14

A computing system comprising:
a processor; and
storage comprising instructions executable by the processor to:
receive a first image from a first camera and a second image from a second camera having an overlapping field of view with the first camera,
detect a skeleton location of a person in the first image and detect the skeleton location of the person in the second image in a region of each image at which fields of view of the first camera and the second camera overlap,
determine a geometric relationship between the cameras of the camera pair based upon comparing coordinates of the skeleton location of the person in the first camera and coordinates of the skeleton location of the person in the second camera,
determine a camera connection graph based on the geometric relationships between the cameras of the camera pairs, and
track an object that moves between fields of view of two or more cameras of the plurality of cameras based upon the coordinate mappings.

### Claim 15 (depends on 14)

The computing system of claim 14, wherein the instructions are further executable to:
determine a primary camera from the camera connection graph,
determine coordinate mappings defining a spatial relationship between the primary camera and each of one or more other cameras of the plurality of cameras, and
track the object based on the coordinate mappings.

### Claim 16 (depends on 14)

The computing system of claim 14, wherein the instructions are further executable to determine a pairwise connection cost for each of the one or more camera pairs, and determine the camera connection graph based on the pairwise connection costs.

### Claim 17 (depends on 14)

The computing system of claim 14, wherein the instructions are executable to determine the pairwise connection cost for each of the one or more camera pairs by applying a 3D rigid transformation matrix using a linear fitting algorithm and/or a RANSAC fitting algorithm.

### Claim 18 (depends on 14)

The computing system of claim 14, wherein the instructions are executable to determine the camera connection graph by applying a minimum spanning tree algorithm based upon the pairwise connection costs.

### Claim 19 (depends on 14)

The computing system of claim 14, wherein the instructions are further executable to divide the camera connection graph into two or more subgraphs based upon the pairwise connection costs.

### Claim 20 (depends on 14)

The computing system of claim 14, wherein the instructions are further executable to determine a ground plane of the space based on locations at which a reference point of the skeletal representation of the person contacted a three-dimensional representation of the space by combining the ground plane estimation in the plurality of cameras.
