# US 10,504,243 B2 · Calibration system for a head-mounted display tracking system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,243 B2 - Document ID: US10504243B2 - Status: Granted - Kind: B2 - Application number: 15/728,391 - Filed: 2017-10-09 - Published: 2019-04-11 - Granted: 2019-12-10 - Earliest claimed: 2017-10-09 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Oskar Linde; Andrew Melim - Web page: https://patentgrep.com/patent/US10504243B2 ## CPC - G06T 7/85: {Stereo camera calibration} - G02B 27/017: {Head mounted} - G06T 7/292: Multi-camera tracking - H04N 13/398: Synchronisation thereof; Control thereof - H04N 23/60: Control of cameras or camera modules - H04N 23/73: by influencing the exposure time - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06T 7/285: using a sequence of stereo image pairs - G06T 2207/30208: Marker matrix - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/246: Calibration of cameras - H04N 13/344: with head-mounted left-right displays - H04N 13/366: using viewer tracking - H04N 17/002: {for television cameras} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A calibration system is configured to determine calibration information of a head-mounted display (HMD). The calibration system comprises a first, second, and third planar grid, a movable platform, and a calibration controller. Each planar grid includes a plurality of fiducial markers that are displayed in accordance with a display pattern. The HMD is coupled to the movable platform, which moves the HMD before the planar grids as a plurality of cameras on the HMD captures images of the planar grids with fiducial markers. The calibration controller controls a motion sequence of the movable platform and determines calibration information for each of the cameras on the HMD and calibration information for an inertial measurement unit (IMU) within the HMD. The calibration information is based in part on a parameterized model of the motion sequence of the HMD. ## Claims ### Claim 1 A calibration system comprising: a first planar grid comprising a first plurality of fiducial markers, the first planar grid positioned at a first angle to an alignment axis; a second planar grid comprising a second plurality of fiducial markers, the second planar grid positioned at a second angle to the alignment axis and a side of the first planar grid and a side of the second planar grid are adjacent to each other; a third planar grid comprising a third plurality of fiducial markers, the third planar grid positioned above the first planar grid and the second planar grid and centered about the alignment axis, a side of the third planar grid adjacent to a portion of the first planar grid and a portion of the second planar grid; a movable platform configured to couple a head-mounted display (HMD), the platform configured to move the HMD positioned before the planar grids as a plurality of cameras on the HMD captures images of the planar grids, the captured images including at least a portion of one of the planar grids with fiducial markers; and a calibration controller configured to control a motion sequence of the movable platform, the calibration controller further configured to determine calibration information for each of the cameras on the HMD and calibration information for an inertial measurement unit (IMU) within the HMD, the calibration information based in part on a parameterized model of the motion sequence of the HMD coupled to the movable platform as the plurality of cameras on the HMD captures images of the planar grids. ### Claim 2 (depends on 1) The calibration system of claim 1, wherein the first planar grid, second planar grid, and third planar grid are electronic displays. ### Claim 3 (depends on 1) The calibration system of claim 1, wherein the first, second, and third pluralities of fiducial markers are each arranged in a grid pattern, and at least one of the fiducial markers in each plurality of fiducial markers is configured to serve as a reference point for a particular location on the respective planar grid. ### Claim 4 (depends on 1) The calibration system of claim 1, wherein the motion sequence of the movable platform resembles a motion of a user wearing the HMD. ### Claim 5 (depends on 1) The calibration system of claim 1, wherein calibration controller is configured to command the first, second, and third planar grids to display the first, second, and third pluralities of fiducial markers in accordance with a display pattern. ### Claim 6 (depends on 5) The calibration system of claim 5, wherein the command includes instructions selected from a group consisting of: a selection of one or more fiducial markers to display, a time point at which to display the one or more selected fiducial markers, and a duration of time to display the one or more selected fiducial markers. ### Claim 7 (depends on 1) The calibration system of claim 1, wherein the calibration controller maps a position of each fiducial marker on the first, second, and third planar grids to a specific location within a captured image. ### Claim 8 (depends on 1) The calibration system of claim 1, wherein the calibration controller is configured to predict IMU information using the parameterized model. ### Claim 9 (depends on 8) The calibration system of claim 8, wherein IMU information is selected from a group consisting of: position information, velocity information, acceleration information, and some combination thereof. ### Claim 10 (depends on 8) The calibration system of claim 8, wherein the calibration controller is configured to predict image information of each of the cameras on the HMD using the parameterized model. ### Claim 11 (depends on 10) The calibration system of claim 10, wherein the image information comprises a portion of the fiducial markers selected from a group consisting of: the first plurality, second plurality, and third plurality of fiducial markers. ### Claim 12 (depends on 11) The calibration system of claim 11, wherein the calibration controller is configured to predict a mapping of the position of each fiducial marker to a specific location within a captured image. ### Claim 13 (depends on 10) The calibration system of claim 10, wherein the calibration controller is further configured to determine calibration information for each of the cameras on the HMD and calibration information for the IMU within the HMD by optimizing the predicted IMU and image information versus the captured IMU and image information captured by the HMD. ### Claim 14 (depends on 1) The calibration system of claim 1, wherein the HMD is heated to a calibration temperature before being coupled to the movable platform for calibration. ### Claim 15 (depends on 14) The calibration system of claim 14, wherein the movable platform is configured to maintain the HMD at the calibration temperature. ### Claim 16 A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: moving a movable platform coupling a head-mounted display (HMD) in accordance with a motion sequence, the movable platform positioned before a plurality of planar grids each comprising a plurality of fiducial markers, the HMD comprising a plurality of cameras configured to capture images of the plurality of planar grids and an inertial measurement unit (IMU) configured to capture measurement signals in response to movement of the HMD; capturing a plurality of images of the planar grids and measurement signals during the motion sequence, wherein each captured image comprises a portion of the plurality of fiducial markers of at least one planar grid; mapping a position of a fiducial marker in a captured image to a specific location on one of the plurality of planar grids; and determining calibration information for each of the cameras on the HMD and calibration information for the IMU. ### Claim 17 (depends on 16) The computer-readable medium of claim 16, the instructions further comprising instructions for sending a command to the plurality of planar grids to display the plurality of fiducial markers in accordance with a display pattern. ### Claim 18 (depends on 17) The computer-readable medium of claim 17, wherein the command is selected from a group consisting of: a selection of one or more fiducial markers to display, a time point at which to display the one or more selected fiducial markers, and a duration of time to display the one or more selected fiducial markers. ### Claim 19 (depends on 16) The computer-readable medium of claim 16, the instructions further comprising instructions for: parameterizing a model of the motion sequence of the HMD coupled to the movable platform; predicting IMU information and image information of each of the cameras on the HMD using the parameterized model; and predicting a mapping of a position of each fiducial marker on each of the plurality of planar grids to a specific location within a captured image. ### Claim 20 (depends on 19) The computer-readable medium of claim 19, wherein determining calibration information for each of the cameras on the HMD and calibration information for the IMU within the HMD comprises optimizing the predicted IMU and image information versus the captured IMU and image information captured by the HMD.