# US 12,223,679 B2 · Data processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,679 B2
- Document ID: US12223679B2
- Status: Granted
- Kind: B2
- Application number: 17/609,492
- Filed: 2020-04-24
- Published: 2022-07-21
- Granted: 2025-02-11
- Earliest claimed: 2019-05-22
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment Inc.
- Inventors: Sharwin Winesh Raghoebardajal; Takanori Minamino
- Web page: https://patentgrep.com/patent/US12223679B2

## CPC

- G06T 7/75: {involving models}
- H04N 13/344: with head-mounted left-right displays
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 15/08: Volume rendering
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 13/398: Synchronisation thereof; Control thereof
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A data processing device includes a plurality of cameras arranged according to a predetermined configuration, where each camera is configured to capture successive images and to detect a current location of one or more elements in the captured images with respect to a coordinate system of the camera, a processor to generate a common camera coordinate system in dependence upon the respective coordinate systems of the plurality of cameras, the common camera coordinate system comprising a set of locations in a three-dimensional space included within the field of view of the plurality of cameras, and a detector to detect a current location of a given element with respect to the common camera coordinate system based on images captured by the plurality of cameras, wherein for each camera, the processor is configured, in response to a detection by the detector that the current location of the given element corresponds to a location in the set of locations of the common camera coordinate system, to generate first data for that location in the set of locations indicative of a mapping between the coordinate system for the camera and the common camera coordinate system, and wherein in response to a detection of a location in the set of locations for which the first data is indicative of the mapping for two or more cameras with respect to the common camera coordinate system, the processor is configured to generate second data for that location indicative of a mapping between the coordinate systems for the two or more cameras.

## Claims

### Claim 1

A data processing apparatus comprising:
a plurality of cameras arranged according to a predetermined configuration, wherein each camera is configured to capture successive images and to detect a current location of one or more elements in the captured images with respect to a coordinate system of the camera;
a processor to generate a common camera coordinate system in dependence upon respective coordinate systems of the plurality of cameras, in which the processor is configured to generate the common camera coordinate system to comprise a generated set of locations including a three-dimensional array of evenly or non-evenly spaced locations in a three-dimensional space included within a field of view of the plurality of cameras, each location of the set of locations having an associated entry that is empty; and
a detector to detect a current location of a given element with respect to the common camera coordinate system based on images captured by the plurality of cameras, wherein
for each camera, the processor is configured, in response to a detection by the detector that the current location of the given element corresponds to a location in the set of locations of the common camera coordinate system, to generate first data for that location in the set of locations indicative of a mapping between the coordinate system for the camera and the common camera coordinate system and populate an associated entry for that location with the first data, and wherein
in response to a detection of a location in the set of locations for which the first data is indicative of the mapping for two or more cameras with respect to the common camera coordinate system, the processor is configured to generate second data for that location indicative of a mapping between the coordinate systems for the two or more cameras and populate an associated entry for that location with the second data.

### Claim 2 (depends on 1)

A data processing apparatus according to claim 1, wherein the processor is configured to generate the first data for each current location of the given element to generate the first data for at least a plurality of locations in the set of locations.

### Claim 3 (depends on 2)

A data processing apparatus according to claim 2, wherein for a location in the set of locations for which the second data is not generated, the processor is configured to derive a mapping between the coordinate systems for two or more cameras for that location based on at least one of the first data and the second data generated for one or more other locations in the set of locations and to generate the second data for that location based on the derivation.

### Claim 4 (depends on 2)

A data processing apparatus according to claim 2, wherein for a location in the set of locations for which the second data is not generated, the processor is configured to derive a mapping between the coordinate systems for two or more cameras for that location based on the second data generated for one or more other locations in the set of locations and to generate the second data for that location based on the derivation.

### Claim 5 (depends on 4)

A data processing apparatus according to claim 4, wherein the processor is configured to derive the mapping between the coordinate systems for the two or more cameras based on an interpolation of the second data generated for the one or more other locations in the set of locations.

### Claim 6 (depends on 2)

A data processing apparatus according to claim 2, wherein the plurality of cameras are configured to repeatedly detect the current locations of one or more of the elements and the processor is configured to generate the first data in response to the detection of the current locations of the one or more elements with respect to the common camera coordinate system.

### Claim 7 (depends on 1)

A data processing apparatus according to claim 1, wherein each location in the set of locations corresponds to a voxel defining a portion of the three-dimensional space included within the field of view of the plurality of cameras.

### Claim 8 (depends on 1)

A data processing apparatus according to claim 1, wherein each camera is configured to detect the location of the given element in the captured images with respect to the coordinate system of the camera based on a model defining a relationship between a point in a captured image and a location with respect to the coordinate system of the camera.

### Claim 9 (depends on 8)

A data processing apparatus according to claim 8, wherein the model comprises one or more camera calibration parameters from among:
focal length;
principal point;
camera position; and
camera orientation.

### Claim 10 (depends on 1)

A data processing apparatus according to claim 1, wherein one or more elements comprise one or more electrically illuminable elements.

### Claim 11 (depends on 10)

A data processing apparatus according to claim 10, wherein the one or more electrically illuminable elements are arranged on a surface of one or more of:
a head mountable display;
a handheld controller;
a game console; and
a light bar.

### Claim 12 (depends on 1)

A data processing apparatus according to claim 1, wherein the plurality of cameras are mounted on or proximate to one or more of:
a head mountable display;
a handheld controller; and
a game console.

### Claim 13

A method comprising:
capturing successive images using a plurality of cameras arranged according to a predetermined configuration and each having a coordinate system;
detecting a current location of one or more elements in the captured images with respect to the coordinate system of a camera;
generating a common camera coordinate system in dependence upon the respective coordinate systems of the plurality of cameras, comprising generating the common camera coordinate system to comprise a generated set of locations including a three-dimensional array of evenly or non-evenly spaced locations in a three-dimensional space included within a field of view of the plurality of cameras, each location of the set of locations having an associated entry that is empty;
detecting a current location of a given element with respect to the common camera coordinate system based on images captured by the plurality of cameras;
for each camera:
generating, in response to detecting that the current location of the given element corresponds to a location in the set of locations of the common camera coordinate system, first data for that location in the set of locations indicative of a mapping between the coordinate system for the camera and the common camera coordinate system and populating an associated entry for that location with the first data; and
generating, in response to a detection of a location in the set of locations for which the first data is indicative of the mapping for two or more cameras with respect to the common camera coordinate system, second data for that location indicative of a mapping between the coordinate systems for the two or more cameras and populating an associated entry for that location with the second data.

### Claim 14

A non-transitory, computer readable storage medium containing computer software which, when executed by a computer, causes the computer to perform a method, comprising:
capturing successive images using a plurality of cameras arranged according to a predetermined configuration and each having a coordinate system;
detecting a current location of one or more elements in the captured images with respect to the coordinate system of a camera;
generating a common camera coordinate system in dependence upon the respective coordinate systems of the plurality of cameras, comprising generating the common camera coordinate system to comprise a generated set of locations including a three-dimensional array of evenly or non-evenly spaced locations in a three-dimensional space included within the field of view of the plurality of cameras, each location of the set of locations having an associated entry that is empty;
detecting a current location of a given element with respect to the common camera coordinate system based on images captured by the plurality of cameras;
for each camera:
generating, in response to detecting that the current location of the given element corresponds to a location in the set of locations of the common camera coordinate system, first data for that location in the set of locations indicative of a mapping between the coordinate system for the camera and the common camera coordinate system and populating an associated entry for that location with the first data; and
generating, in response to a detection of a location in the set of locations for which the first data is indicative of the mapping for two or more cameras with respect to the common camera coordinate system, second data for that location indicative of a mapping between the coordinate systems for the two or more cameras and populating an associated entry for that location with the second data.