# US 10,503,247 B2 · Calibration of stereo cameras and handheld object

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,503,247 B2
- Document ID: US10503247B2
- Status: Granted
- Kind: B2
- Application number: 15/826,575
- Filed: 2017-11-29
- Published: 2018-11-15
- Granted: 2019-12-10
- Earliest claimed: 2017-05-09
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Michael Edward Samples; Alexandru Octavian Balan; Salim Sirtkaya; Vuk Jovanovic; Filip Panjevic; Taras Khapko; Ruven Jaime Rivera, III
- Web page: https://patentgrep.com/patent/US10503247B2

## CPC

- G06F 3/012: {Head tracking input arrangements}
- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/212: using sensors worn by the player, e.g. for measuring heart beat or leg activity
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/22: Setup operations, e.g. calibration, key configuration or button assignment
- A63F 13/235: using a wireless connection, e.g. infrared or piconet
- A63F 13/24: Constructional details thereof, e.g. game controllers with detachable joystick handles
- A63F 13/25: Output arrangements for video game devices
- A63F 13/31: Communication aspects specific to video games, e.g. between several handheld game devices at close range
- A63F 13/42: by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- A63F 13/5255: according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
- A63F 13/53: involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
- A63F 13/80: Special adaptations for executing a specific game genre or game mode
- A63F 13/92: Video game devices specially adapted to be hand-held while playing
- F21V 23/0471: {the sensor detecting the proximity, the presence or the movement of an object or a person}
- G01B 11/002: {for measuring two or more coordinates}
- G01C 19/5776: Signal processing not specific to any of the devices covered by groups G01C19/5607 - G01C19/5719
- G01D 5/26: characterised by optical transfer means, i.e. using infrared, visible, or ultraviolet light
- G01J 1/44: Electric circuits {(for command of an exposure part G03B7/02)}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0308: {comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen}
- G06F 3/0325: {using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image}
- G06F 3/033: Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041)
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 7/248: {involving reference images or patches}
- G06T 7/285: using a sequence of stereo image pairs
- G06T 7/73: using feature-based methods
- G06T 7/74: {involving reference images or patches}
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 7/85: {Stereo camera calibration}
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/246: Calibration of cameras
- H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects
- H05B 45/10: Controlling the intensity of the light
- H05B 47/11: by determining the brightness or colour temperature of ambient light
- H05B 47/125: by using cameras
- H05B 47/19: via wireless transmission
- H05B 47/1965: {using handheld communication devices}
- A63F 2300/8082: Virtual reality
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06F 2203/0382: Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10152: Varying illumination
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/30244: Camera pose
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- Y02B 20/40: Control techniques providing energy savings, e.g. smart controller or presence detection
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- F21V: FUNCTIONAL FEATURES OR DETAILS OF LIGHTING DEVICES OR SYSTEMS THEREOF; STRUCTURAL COMBINATIONS OF LIGHTING DEVICES WITH OTHER ARTICLES, NOT OTHERWISE PROVIDED FOR
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01D: MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
- G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H05B: ELECTRIC HEATING; ELECTRIC LIGHT SOURCES NOT OTHERWISE PROVIDED FOR; CIRCUIT ARRANGEMENTS FOR ELECTRIC LIGHT SOURCES, IN GENERAL
- Y02B: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS

## Abstract

One disclosed example provides a head-mounted device including a stereo camera arrangement, a logic device configured to execute instructions, and a storage device storing instructions executable by the logic device to, for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect light sources of a handheld object in the image data, and based upon the light sources detected, determine a pose of the handheld object. The instructions are executable to, based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.

## Claims

### Claim 1

A head-mounted device, comprising:
a stereo camera arrangement;
a logic device configured to execute instructions; and
a storage device comprising instructions executable by the logic device to
for each camera in the stereo camera arrangement,
receive image data of a field of view of the camera,
detect a plurality of light sources of a handheld object in the image data, the plurality of light sources comprising a subset of light sources of the handheld object, and
based upon the plurality of light sources detected, determine a pose of the handheld object; and
based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.

### Claim 2 (depends on 1)

The head-mounted device of claim 1, wherein the instructions are executable to calibrate the stereo camera arrangement during one or more of a motion of the head-mounted device and a motion of the handheld object.

### Claim 3 (depends on 1)

The head-mounted device of claim 1, wherein the instructions are executable to calibrate intrinsic characteristics for each camera of the stereo camera arrangement, the intrinsic characteristics comprising one or more of lens distortion, focal length, and principal point.

### Claim 4 (depends on 1)

The head-mounted device of claim 1, wherein the instructions are executable to calibrate extrinsic characteristics for each camera of the stereo camera arrangement, the extrinsic characteristics comprising one or more of a rotation and a translation.

### Claim 5 (depends on 1)

The head-mounted device of claim 1, wherein the instructions are executable to determine the pose by fitting an observed light pattern to a model light pattern via a rigid body transformation.

### Claim 6 (depends on 1)

The head-mounted device of claim 1, wherein the instructions are executable to determine the pose by, for each light source detected in the image data received from a first camera, searching along an epipolar line for a companion pixel in the image data received from a second camera, the companion pixel corresponding to the light source detected in the image data received from the first camera.

### Claim 7 (depends on 1)

The head-mounted device of claim 1, wherein the instructions are executable to calibrate the stereo camera arrangement utilizing one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.

### Claim 8 (depends on 1)

The head-mounted device of claim 1, wherein the instructions are executable to calibrate the stereo camera arrangement between 6 and 10 times per minute.

### Claim 9

A method for determining a calibration for a handheld object via a head-mounted device comprising a stereo camera arrangement, the method comprising:
receiving, from a first camera of the stereo camera arrangement, first image data from a perspective of the first camera;
receiving, from a second camera of the stereo camera arrangement, second image data from a perspective of the second camera;
detecting, in the first image data and the second image data, a plurality of light sources of the handheld object;
determining a three-dimensional position of each light source of the plurality of light sources detected; and
determining a calibration for the handheld object based upon each three-dimensional position determined.

### Claim 10 (depends on 9)

The method of claim 9, further comprising
receiving from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and
determining a calibration for the IMU based upon the sensed orientation.

### Claim 11 (depends on 10)

The method of claim 10, wherein determining the calibration for the IMU comprises determining a calibration for a position and a rotation of the IMU.

### Claim 12 (depends on 9)

The method of claim 9, wherein determining the calibration of the handheld object comprises determining calibration parameters via one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.

### Claim 13 (depends on 9)

The method of claim 9, wherein determining the three-dimensional position of each light source comprises fitting an observed light pattern to a model light pattern via a rigid body transformation.

### Claim 14

A head-mounted device, comprising:
a stereo camera arrangement;
a logic device configured to execute instructions; and
a storage device comprising instructions executable by the logic device to
for each camera in the stereo camera arrangement,
receive image data of a surrounding environment from a perspective of the camera,
detect via the image data a plurality of feature points of a handheld object in a field of view of the camera,
determine a location of each feature point detected, and
determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined;
update one or more calibration parameters for the stereo camera arrangement based on the pose of the handheld object determined for each camera;
determine a three-dimensional position of each feature point based on the location of each feature point determined for each camera of the stereo camera arrangement; and
update one or more calibration parameters for the handheld object based on the three-dimensional position of each feature point determined.

### Claim 15 (depends on 14)

The head-mounted device of claim 14, wherein the instructions are executable to receive the image data as a background process during use of the head-mounted device and the handheld object.

### Claim 16 (depends on 14)

The head-mounted device of claim 14, wherein the instructions are further executable to receive from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and to determine a calibration for one or more of a translation and a rotation of the IMU based upon the sensed orientation.

### Claim 17 (depends on 14)

The head-mounted device of claim 14, wherein the one or more calibration parameters for the stereo camera arrangement comprise one or more of an intrinsic camera characteristic and an extrinsic camera characteristic.

### Claim 18 (depends on 14)

The head-mounted device of claim 14, wherein the instructions are executable to determine the pose of the handheld object by one or more of
fitting an observed feature point pattern to a model feature point pattern via a rigid body transformation; and
for each feature point detected in the image data received from a first camera of the stereo camera arrangement, searching along an epipolar line for a companion feature point in the image data received from a second camera of the stereo camera arrangement, the companion feature point corresponding to the feature point detected in the image data received from the first camera.

### Claim 19 (depends on 14)

The head-mounted device of claim 14, wherein the instructions are executable to update the one or more calibration parameters for the stereo camera arrangement and update the one or more calibration parameters for the handheld object simultaneously.

### Claim 20 (depends on 14)

The head-mounted device of claim 14, wherein the instructions are executable to update one or more calibration parameters for one or more of the stereo camera arrangement and the handheld object via one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.