# US 2019/0174122 A1 · Method, system and apparatus for capture of image data for free viewpoint video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0174122 A1
- Document ID: US20190174122A1
- Status: Published application
- Kind: A1
- Application number: 15/831,193
- Filed: 2017-12-04
- Published: 2019-06-06
- Earliest claimed: 2017-12-04
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: James Austin Besley
- Web page: https://patentgrep.com/patent/US20190174122A1

## CPC

- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- G06T 7/85: {Stereo camera calibration}
- G06T 15/20: Perspective computation
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/139: Format conversion, e.g. of frame-rate or size
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/246: Calibration of cameras
- H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for synthesising a viewpoint, comprising: capturing a scene using a network of cameras, the cameras defining a system volume of the scene, wherein a sensor of one of the cameras has an output frame rate for the system volume below a predetermined frame rate; selecting a portion of the system volume as an operational volume based on the sensor output frame rate, the predetermined frame rate and a region of interest, the operational volume being a portion of the system volume from which image data for the viewpoint can be synthesised at the predetermined frame rate, wherein a frame rate for synthesising a viewpoint outside the operational volume is limited by the output frame rate; reading, from the sensors at the predetermined frame rate, image data corresponding to the operational volume; and synthesising the viewpoint at the predetermined frame rate using the image data.

## Claims

### Claim 1

A method for synthesising a viewpoint at a predetermined frame rate, the method comprising:
capturing a scene using a network comprising a plurality of image capture devices, the plurality of image capture devices defining a system volume of the scene, wherein one of the sensors of at least one of the image capture devices has an output frame rate for the system volume below the predetermined frame rate;
selecting a portion of the system volume as an operational volume based on the sensor output frame rate, the predetermined frame rate and a region of interest in the scene, the operational volume being a portion of the system volume from which image data for the viewpoint can be synthesised at the predetermined frame rate, wherein a frame rate for synthesising a viewpoint outside the operational volume is limited by the sensor output frame rate associated with at least one of the capture devices;
reading, from the sensors and at the predetermined frame rate, image data corresponding to the operational volume; and
synthesising the viewpoint at the predetermined frame rate using the image data, the viewpoint being synthesised by placing a virtual camera within the operational volume based on viewpoint parameters.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the image data corresponding to the operational volume read from each sensor corresponds to a group of pixels having a geometry different to the geometry of the sensor.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the group of pixels has a different aspect ratio to the sensor and the group of pixels has a different shape to the sensor.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising tracking the region of interest in the image data read from the sensors and updating the operational volume based on the tracking.

### Claim 5 (depends on 1)

The method according to claim 1, further comprising determining the region of interest in the scene based on image data captured by the plurality of image capture devices.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the region of interest is selected by an operator of the network of image capture devices.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising determining the region of interest in the scene based on image data captured by the plurality of image capture devices and mapping pixels of the sensor of each of the image capture devices to the region of interest.

### Claim 8 (depends on 1)

The method according to claim 1, further comprising reading alignment data from the sensors, the alignment data relating to regions of the scene outside the region of interest.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising determining the alignment regions from image data captured for the scene by the plurality of image capture devices and performing patch-based alignment using the alignment data and calibrating the network of cameras based on the patch-based alignment.

### Claim 10 (depends on 1)

The method according to claim 1, further comprising reading alignment data from the sensors, the alignment data relating to regions of the scene outside the region of interest determined in advance and performing patch-based alignment using the alignment data and calibrating the network of cameras based on the patch-based alignment.

### Claim 11 (depends on 8)

The method according to claim 8, further comprising performing patch-based alignment using the alignment data and updating the pixels read out from the sensor based on the alignment.

### Claim 12 (depends on 1)

The method according to claim 1, further comprising determining background pixels of the operational volume using a background model, and reading out the determined background pixels at a frame rate below the predetermined frame rate.

### Claim 13 (depends on 1)

The method according to claim 1, further comprising reading out pixels forming a border a predetermined width around the operation volume, the pixels forming the border being read out at a frame rate below the predetermined frame rate.

### Claim 14

A non-transitory computer-readable medium having a program stored thereon for synthesising a viewpoint at a predetermined frame rate, the program comprising:
code for capturing a scene using a network comprising a plurality of image capture devices, the plurality of image capture devices defining a system volume of the scene, wherein one of the sensors of at least one of the image capture devices has an output frame rate for the system volume below the predetermined frame rate;
code for selecting a portion of the system volume as an operational volume based on the sensor output frame rate, the predetermined frame rate and a region of interest in the scene, the operational volume being a portion of the system volume from which image data for the viewpoint can be synthesised at the predetermined frame rate, wherein a frame rate for synthesising a viewpoint outside the operational volume is limited by the sensor output frame rate associated with at least one of the capture devices;
code for reading, from the sensors and at the predetermined frame rate, image data corresponding to the operational volume; and
code for synthesising the viewpoint at the predetermined frame rate using the image data, the viewpoint being synthesised by placing a virtual camera within the operational volume based on viewpoint parameters.

### Claim 15

Apparatus configured to:
capture a scene using a network comprising a plurality of image capture devices, the plurality of image capture devices defining a system volume of the scene, wherein one of the sensors of at least one of the image capture devices has an output frame rate for the system volume below the predetermined frame rate;
select a portion of the system volume as an operational volume based on the sensor output frame rate, the predetermined frame rate and a region of interest in the scene, the operational volume being a portion of the system volume from which image data for the viewpoint can be synthesised at the predetermined frame rate, wherein a frame rate for synthesising a viewpoint outside the operational volume is limited by the sensor output frame rate associated with at least one of the capture devices;
read, from the sensors and at the predetermined frame rate, image data corresponding to the operational volume; and
synthesise the viewpoint at the predetermined frame rate using the image data, the viewpoint being synthesised by placing a virtual camera within the operational volume based on viewpoint parameters.

### Claim 16

A system, comprising:
a network comprising a plurality of image capture devices, the plurality of image capture devices defining a system volume of a scene, wherein one of the sensors of at least one of the image capture devices has an output frame rate for the system volume below a predetermined frame rate;
a memory for storing data and a computer readable medium;
a processor coupled to the memory for executing a computer program, the program having instructions for:
capturing the scene using the network comprising the plurality of image capture devices;
selecting a portion of the system volume as an operational volume based on the sensor output frame rate, the predetermined frame rate and a region of interest in the scene, the operational volume being a portion of the system volume from which image data for the viewpoint can be synthesised at the predetermined frame rate, wherein a frame rate for synthesising a viewpoint outside the operational volume is limited by the sensor output frame rate associated with at least one of the capture devices;
reading, from the sensors and at the predetermined frame rate, image data corresponding to the operational volume; and
synthesising the viewpoint at the predetermined frame rate using the image data, the viewpoint being synthesised by placing a virtual camera within the operational volume based on viewpoint parameters

### Claim 17 (depends on 16)

The system according to claim 16, wherein the program further comprises instructions for tracking the region of interest in the image data read from the sensors and updating the operational volume based on the tracking.

### Claim 18 (depends on 16)

The system according to claim 16, wherein the program further comprises instructions for reading alignment data from the sensor, the alignment data relating to regions of the scene outside the region of interest.

### Claim 19 (depends on 18)

The system according to claim 18, wherein the program further comprises instructions for calibrating the network using the alignment data and updating the pixels read out from the sensor based on the calibration.

### Claim 20 (depends on 16)

The system according to claim 16, wherein the program further comprises instructions for determining background pixels of the operational volume using a background model derived based on pixels outside the region of interest.
