# US 11,308,644 B2 · Multi-presence detection for performance capture > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,644 B2 - Document ID: US11308644B2 - Status: Granted - Kind: B2 - Application number: 17/378,595 - Filed: 2021-07-16 - Published: 2022-03-03 - Granted: 2022-04-19 - Earliest claimed: 2020-08-28 - Assignees: Weta Digital Limited - Applicants: Weta Digital Limited - Inventors: Dejan Momcilovic; Jake Botting - Web page: https://patentgrep.com/patent/US11308644B2 ## CPC - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/73: using feature-based methods - H04N 17/002: {for television cameras} - G06T 2207/30196: Human being; Person - G06T 2207/30204: Marker - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present description relates relate to recalibration of a sensor device for performance capture, by detecting a miscalibration problem with sensor device and assessment of the problem. A recalibration system includes sensor devices initially calibrated at a recording site. A recording site change occurs and afterwards, a failure to match virtual rays projected from one sensor device with virtual rays projected from the active marker is detected. In response to determining the failure, the active marker is signaled to emit a unique display of light. The failure of the rays to match is assessed based on whether sensor devices capture the unique display of light. Three-dimensional (3-D) coordinates of an active marker is reconstructed from marker data of the calibrated sensor devices. A problematic sensor device is recalibrated based on the assessment, using the 3-D coordinates of the active marker from marker data of the remaining calibrated sensor devices, without stopping the recording. ## Claims ### Claim 1 A method for recalibrating a sensor device to capture light from an active marker during a recording for a virtual production, the method comprising: providing a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker; after a recording site change, determining a failure of virtual rays projected from the first sensor device to match with virtual rays projected from the active marker; in response to determining the failure, signaling for the active marker to emit a unique display of light; assessing the failure based on at least one of the first sensor device, the second sensor device, or the third sensor device capturing the unique display of light; and without stopping the recording, recalibrating the first sensor device based on the assessment, using 3-D coordinates of the active marker from marker data of the second sensor device and the third sensor device. ### Claim 2 (depends on 1) The method of claim 1, wherein assessing the failure includes: detecting that the first sensor device, the second sensor device, and the third sensor device each capture the unique display of light; and determining that the first sensor device is miscalibrated and the active marker is in a field of view of the first sensor device. ### Claim 3 (depends on 2) The method of claim 2, wherein the recording site change includes the first sensor device being moved to a changed physical position. ### Claim 4 (depends on 3) The method of claim 3, wherein the recalibrating includes calculating the changed physical position of the first sensor device based by applying the 3-D coordinates of the active marker and corresponding two-dimensional (2-D) coordinates of the first sensor device to a perspective-n-point pose problem formula. ### Claim 5 (depends on 1) The method of claim 1, wherein assessing the failure includes: detecting that the second sensor device or the third sensor device captures the unique display of light and the first sensor device fails to capture the unique display of light; and repositioning the first sensor device to a changed physical position to have a field of view region including the active marker and overlapping with a field of view of the second sensor device or the third sensor device, thereby causing the first sensor device to be miscalibrated. ### Claim 6 (depends on 5) The method of claim 5, wherein the recording site change includes an initial field of view of the first sensor device in an initial sensor position becoming at least partially occluded. ### Claim 7 (depends on 1) The method of claim 1, wherein the recording site change is a modification of an intrinsic parameter of the first sensor device. ### Claim 8 (depends on 1) The method of claim 1, wherein assessing the failure further includes: identifying the active marker by the marker data representing the unique display of light emitted by the active marker and captured in particular frames by the first sensor device, the second sensor device, and the third sensor device, when each sensor device is initially calibrated. ### Claim 9 A system for recalibrating a sensor device to capture active marker light during a recording associated with a virtual production, the system comprising: a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker; a plurality of active markers; at least one computing device comprising one or more processors and a tangible processor-readable storage device including instructions for: after a recording site change, determining a failure of virtual rays projected from the first sensor device to match with virtual rays projected from the active marker; in response to determining the failure, signaling for the active marker to emit a unique display of light; assessing the failure based on at least one of the first sensor device the second sensor device, or the third sensor device capturing the unique display of light; and without stopping the recording, recalibrating the first sensor device based on the assessment, using 3-D coordinates of the active marker from marker data of the second sensor device and the third sensor device. ### Claim 10 (depends on 9) The system of claim 9, wherein assessing the failure includes: detecting the first sensor device, the second sensor device, and the third sensor device each capture the unique display of light; and determining that the first sensor device is miscalibrated and the active marker is in a field of view of the first sensor device. ### Claim 11 (depends on 9) The system of claim 9, wherein the recording site change includes the first sensor device being moved to a changed physical position, and wherein the recalibrating includes calculating the changed physical position of the first sensor device based by applying the 3-D coordinates of the active marker and corresponding two-dimensional (2-D) coordinates of the first sensor device to a perspective-n-point pose problem formula. ### Claim 12 (depends on 9) The system of claim 9, wherein the recording site change includes a field of view of the first sensor device in an initial sensor position becoming at least partially occluded and wherein assessing the failure further comprises: detecting that the second sensor device or the third sensor device captures the unique display of light and the first sensor device fails to capture the unique display of light; and repositioning the first sensor device to a changed physical position to have a field of view region including the active marker and overlapping with a field of view of the second sensor device or the third sensor device, thereby causing the first sensor device to be miscalibrated. ### Claim 13 (depends on 9) The system of claim 9, wherein the recording site change is a modification of an intrinsic parameter of the first sensor device. ### Claim 14 (depends on 9) The system of claim 9, wherein assessing the failure further includes: identifying the active marker by the marker data representing the unique display of light emitted by the active marker and captured in particular frames by the first sensor device and second sensor device. ### Claim 15 A non-transitory computer-readable storage medium carrying program instructions thereon for recalibrating a sensor device to capture active marker light during a recording associated with a virtual production, the program instructions when executed by one or more processors cause the one or more processors to perform operations comprising: receiving initial calibration information for a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker; after a recording site change, determining a failure of virtual rays projected from the first sensor device to match with virtual rays projected from the active marker; in response to determining the failure, signaling for the active marker to emit a unique display of light; assessing the failure based on at least one of the first sensor device, the second sensor device, or the third sensor device capturing the unique display of light; and without stopping the recording, recalibrating the first sensor device based on the assessment, using 3-D coordinates of the active marker from marker data of the second sensor device and the third sensor device. ### Claim 16 (depends on 15) The computer-readable storage medium of claim 15, wherein assessing the failure includes: detecting the first sensor device, the second sensor device, and the third sensor device each capture the unique display of light; and determining that the first sensor device is miscalibrated and the active marker is in a field of view of the first sensor device. ### Claim 17 (depends on 15) The computer-readable storage medium of claim 15, wherein the recording site change includes the first sensor device being moved to a changed physical position, and wherein the recalibrating includes calculating the changed physical position of the first sensor device based by applying the 3-D coordinates of the active marker and corresponding two-dimensional (2-D) coordinates of the first sensor device to a perspective-n-point pose problem formula. ### Claim 18 (depends on 15) The computer-readable storage medium of claim 15, wherein the recording site change includes a field of view of the first sensor device in an initial sensor position becoming at least partially occluded and wherein assessing the failure further comprises: detecting that the second sensor device or the third sensor device captures the unique display of light and the first sensor device fails to capture the unique display of light; and repositioning the first sensor device to a changed physical position to have a field of view region including the active marker and overlapping with a field of view of the second sensor device or the third sensor device, thereby causing the first sensor device to be miscalibrated. ### Claim 19 (depends on 15) The computer-readable storage medium of claim 15, wherein the recording site change is a modification of an intrinsic parameter of the first sensor device. ### Claim 20 (depends on 15) The computer-readable storage medium of claim 15, wherein assessing the failure further includes: identifying the active marker by the marker data representing the unique display of light emitted by the active marker and captured in particular frames by the first sensor device, the second sensor device, and the third sensor device, when each sensor device is initially calibrated.