# US 2013/0278779 A1 · Automatic calibration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0278779 A1
- Document ID: US20130278779A1
- Status: Published application
- Kind: A1
- Application number: 13/451,005
- Filed: 2012-04-19
- Published: 2013-10-24
- Earliest claimed: 2012-04-19
- Assignees: Hewlett-Packard Development Company, L.P.
- Inventors: Wei Hong
- Web page: https://patentgrep.com/patent/US20130278779A1

## CPC

- H04N 7/144: {camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact}
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- H04N 9/3179: {Video signal processing therefor}
- H04N 9/3194: {including sensor feedback}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An automatic calibration method for a projector-camera system including a semi-transparent screen is disclosed herein. An image sequence is caused to be captured from the semi-transparent screen and through the semi-transparent screen while a calibration pattern having features is displayed and not displayed in an alternating succession on the semi-transparent screen. A temporal correlation image is created from the image sequence and a discrete binary signal. Peaks are identified in a spatial cross correlation image generated from the temporal correlation image, where a pattern of the identified peaks corresponds to a pattern of the features in the calibration pattern. The peaks are transformed to coordinates of corrected feature points. A comparison of the corrected feature points and a ground truth set of coordinates for the features is used to determine whether the projector-camera system is calibrated.

## Claims

### Claim 1

An automatic calibration method for a projector-imaging device system including a semi-transparent screen, the method comprising:
causing an image sequence to be captured from the semi-transparent screen and through the semi-transparent screen while a calibration pattern having features is displayed and not displayed in an alternating succession on the semi-transparent screen;
creating a temporal correlation image from the image sequence and a discrete binary signal;
identifying peaks in a spatial cross correlation image generated from the temporal correlation image, where a pattern of the identified peaks corresponds to a pattern of the features in the calibration pattern;
transforming the peaks to coordinates of corrected feature points; and
determining whether the projector-camera system is calibrated by comparing the corrected feature points with a ground truth set of coordinates for the features.

### Claim 2 (depends on 1)

The automatic calibration method as defined in claim 1, further comprising creating the calibration pattern by:
selecting the pattern of the features as M×N;
selecting a 2×2 black-and-white checkerboard as each feature; and
evenly distributing the features at (xi, yj) where i=1 M and j=1 . . . N.

### Claim 3 (depends on 1)

The automatic calibration method as defined in claim 1 wherein the transforming of the peaks to the coordinates of the corrected feature points includes:
identifying detected feature points for each of the peaks by sorting v coordinates of each of the peaks along a y axis and then sorting u coordinates of each of the peaks along an x axis; and
using a transform function, transforming the detected feature points to the corrected feature points.

### Claim 4 (depends on 3)

The automatic calibration method as defined in claim 3 wherein the detected feature points are in a raster scan order that matches the pattern of the features in the calibration pattern.

### Claim 5 (depends on 3)

The automatic calibration method as defined in claim 3 wherein the transform function is a homography.

### Claim 6 (depends on 1)

The automatic calibration method as defined in claim 1 wherein the determining includes one of:
determining calibration success when a root mean square error between the corrected feature points and the ground truth set of coordinates for the features is equal to or smaller than a threshold value; or
determining calibration failure when a root mean square error between the corrected feature points and the ground truth set of coordinates for the features is larger than a threshold value.

### Claim 7 (depends on 1)

The automatic calibration method as defined in claim 1 wherein the creating of the temporal correlation image from the image sequence and the discrete binary signal is accomplished by:
i) representing each image in the image sequence as:
Ik(x,y)=Bk(x,y)+P(x,y)Sk
wherein:
k=1 . . . 2K;
2K is a predetermined number of images in the image sequence;
Bk(x, y) is representative of a background image;
P(x, y) is representative of a calibration pattern image; and
Sk is the discrete binary signal which equals 1 for all odd k and 0 for all even k; and
ii) creating the temporal correlation image, C(x, y), as follows:
=
∑
k
=
1
K

(
B
2


k
-
1

(
x
,
y
)
-
B
2


k

(
x
,
y
)
)
+
KP

(
x
,
y
)

### Claim 8 (depends on 7)

The automatic calibration method as defined in claim 7, further comprising generating the spatial cross correlation image, R(x, y), using an integral image, G(x, y), of the temporal correlation image, C(x, y), as follows:
R(x,y)=2G(x,y)+G(x−d,y−d)+G(x+d,y+d)−G(x−d,y)−G(x,y+d)−G(x,y−d)−G(x+d,y)
wherein d is a side length of one of the features in the calibration pattern.

### Claim 9

A non-transitory, tangible computer readable medium having instructions embedded thereon that, when executed, implement a method for automatically calibrating a projector-imaging device system including a semi-transparent screen, the method comprising:
causing an image sequence to be captured from the semi-transparent screen and through the semi-transparent screen while a calibration pattern having features is displayed and not displayed in an alternating succession on the semi-transparent screen;
creating a temporal correlation image from the image sequence and a discrete binary signal;
identifying peaks in a spatial cross correlation image generated from the temporal correlation image, where a pattern of the identified peaks corresponds to a pattern of the features in the calibration pattern;
transforming the peaks to coordinates of corrected feature points; and
determining whether the projector-camera system is calibrated by comparing the corrected feature points with a ground truth set of coordinates for the features.

### Claim 10 (depends on 9)

The non-transitory, tangible computer readable medium as defined in claim 9 wherein:
the creating of the temporal correlation image from the image sequence and the discrete binary signal is accomplished by:
i) representing each image and each other image as:
Ik(x,y)=Bk(x,y)+P(x,y)Sk
wherein:
k=1 . . . 2K;
2K is a predetermined number of images in the image sequence;
Bk(x, y) is representative of a background image;
P(x, y) is representative of a calibration pattern image; and
Sk is the discrete binary signal which equals 1 for all odd k and 0 for all even k; and
ii) creating the temporal correlation image, C(x, y), as follows:
=
∑
k
=
1
K

(
B
2


k
-
1

(
x
,
y
)
-
B
2


k

(
x
,
y
)
)
+
KP

(
x
,
y
)
and the method further comprises generating the spatial cross correlation image, R(x, y), using an integral image, G(x, y), of the temporal correlation image, C(x, y), as follows:
R(x,y)=2G(x,y)+G(x−d,y−d)+G(x+d,y+d)−G(x−d,y)−G(x,y+d)−G(x,y−d)−G(x+d,y)
wherein d is ½ a side length of one of the features in the calibration pattern.

### Claim 11 (depends on 10)

The non-transitory, tangible computer readable medium as defined in claim 10 wherein the peaks in the spatial cross correlation image represent detected centers of the features of the calibration pattern in the temporal correlation image, and wherein the transforming of the peaks to the coordinates of the corrected feature points includes:
identifying detected feature points for each of the peaks by sorting v coordinates of each of the peaks along a y axis and then sorting u coordinates of each of the peaks along an x axis; and
using a homography transform function, transforming the detected feature points to the corrected feature points.

### Claim 12 (depends on 9)

The non-transitory, tangible computer readable medium as defined in claim 9 wherein the determining includes one of:
determining calibration success when a root mean square error between the corrected feature points and the ground truth set of coordinates for the features is equal to or smaller than a threshold value; or
determining calibration failure when a root mean square error between the corrected feature points and the ground truth set of coordinates for the features is larger than a threshold value.

### Claim 13

An automatic calibration system for a local system of a remote collaboration system, the automatic calibration system including:
an image capturing engine to:
cause an image sequence to be captured from a semi-transparent screen and through the semi-transparent screen of the local system while a calibration pattern having features is displayed and not displayed in an alternating succession on the semi-transparent screen; and
a calibration engine to:
create a temporal correlation image from the image sequence and a discrete binary signal;
identify peaks in a spatial cross correlation image generated from the temporal correlation image, where a pattern of the identified peaks corresponds to a pattern of the features in the calibration pattern;
transform the peaks to coordinates of corrected feature points; and
compare the corrected feature points with a ground truth set of coordinates for the features to determine whether the local system is calibrated.

### Claim 14 (depends on 13)

The automatic calibration system as defined in claim 13 wherein the calibration engine is to further:
create the temporal correlation image from the image sequence and the discrete binary signal by:
i) representing each image and each other image as:
Ik(x,y)=Bk(x,y)+P(x,y)Sk
wherein:
k=1 . . . 2K;
2K is a predetermined number of images in the image sequence;
Bk(x, y) is representative of a background image;
P(x, y) is representative of a calibration pattern image; and
Sk is the discrete binary signal which equals 1 for all odd k and 0 for all even k; and
ii) creating the temporal correlation image, C(x, y), as follows:
=
∑
k
=
1
K

(
B
2


k
-
1

(
x
,
y
)
-
B
2


k

(
x
,
y
)
)
+
KP

(
x
,
y
)
generate the spatial cross correlation image, R(x, y), using an integral image, G(x, y), of the temporal correlation image, C(x, y), as follows:
R(x,y)=2G(x,y)+G(x−d,y−d)+G(x+d,y+d)−G(x−d,y)−G(x,y+d)−G(x,y−d)−G(x+d,y)
wherein d is ½ a side length of one of the features in the calibration pattern.

### Claim 15 (depends on 14)

The automatic calibration system as defined in claim 14 wherein the peaks in the spatial cross correlation image represent detected centers of the features of the calibration pattern in the temporal correlation image, and wherein the transformation of the peaks to the coordinates of the corrected feature points is accomplished by:
identifying detected feature points for each of the peaks by sorting y coordinates of each of the peaks along a y axis and then sorting x coordinates of each of the peaks along an x axis; and
using a homography transform function, transforming the detected feature points to the corrected feature points.