# US 10,437,342 B2 · Calibration systems and methods for depth-based interfaces with disparate fields of view > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,437,342 B2 - Document ID: US10437342B2 - Status: Granted - Kind: B2 - Application number: 15/369,799 - Filed: 2016-12-05 - Published: 2018-06-07 - Granted: 2019-10-08 - Earliest claimed: 2016-12-05 - Assignees: YouSpace, Inc. - Applicants: YouSpace, Inc. - Inventors: Ralph Brunner - Web page: https://patentgrep.com/patent/US10437342B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 7/50: Depth or shape recovery - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06F 2203/0382: Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20: Special algorithmic details - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Various of the disclosed embodiments provide Human Computer Interfaces (HCI) that incorporate depth sensors at multiple positions and orientations. The depth sensors may be used in conjunction with a display screen to permit users to interact dynamically with the system, e.g., via gestures. Calibration methods for orienting depth values between sensors are also presented. The calibration methods may generate both rotation and translation transformations that can be used to determine the location of a depth value acquired in one sensor from the perspective of another sensor. The calibration process may itself include visual feedback to direct a user assisting with the calibration. In some embodiments, floor estimation techniques may be used alone or in conjunction with the calibration process to facilitate data processing and gesture identification. ## Claims ### Claim 1 A display structure for gesture interactions comprising: a display screen; a plurality of sensors, at least two of the plurality of sensors configured to acquire depth information and at least two of the plurality of sensors configured to acquire pixel image information in a region before the display screen; a computer system configured to perform a method comprising: capturing a first pixel image at a primary sensor of at least a portion of the primary sensor's field of view; capturing a second pixel image at a first secondary sensor of at least a portion of the first secondary sensor's field of view; causing the display screen to display the first pixel image; causing the display screen to display the second pixel image; capturing a first set of depth values associated with a calibration object at the primary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; capturing a second set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; determining a first normal associated with a plane of the calibration object for the first set of depth values; determining a first on-plane point associated with the plane of the calibration object for the first set of depth values; determining a second normal associated with the plane of the calibration object for the second set of depth values; determining a second on-plane point associated with the plane of the calibration object for the second set of depth values; generating a first system of linear equations, in part, using a linear equation derived from the first normal and the second normal; solving for components of a rotation transform using the first system of linear equations; generating a second system of linear equations, in part, using a linear equation derived from the first on-plane point and the second on-plane point; and solving for components of a translation transform using the second system of linear equations, wherein the rotation transform and translation transform move a point from the perspective of the first secondary sensor to the perspective of the primary sensor. ### Claim 2 (depends on 1) The display structure of claim 1, wherein the first on-plane point is an average of the depth values associated with the plane visible to the primary sensor. ### Claim 3 (depends on 1) The display structure of claim 1, wherein the calibration object is a planar board. ### Claim 4 (depends on 1) The display structure of claim 1, wherein the primary sensor's field of view and the first secondary sensor's field of view do not overlap. ### Claim 5 (depends on 1) The display structure of claim 1, the method further comprising: capturing a third set of depth values associated with the calibration object at a second secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; capturing a fourth set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; determining a third normal associated with a plane of the calibration object for the third set of depth values; determining a third on-plane point associated with the plane of the calibration object for the third set of depth values; determining a fourth normal associated with a plane of the calibration object for the fourth set of depth values; and determining a fourth on-plane point associated with the plane of the calibration object for the fourth set of depth values, wherein generating the first system of linear equations further comprises inserting a linear equation derived from the third normal and the fourth normal, and wherein generating the second system of linear equations further comprises inserting a linear equation derived from the third on-plane point and the fourth on-plane point. ### Claim 6 (depends on 5) The display structure of claim 5, wherein at least two of the plurality of sensors are vertically offset relative to one another by a bracket. ### Claim 7 A computer-implemented method for calibrating a plurality of sensors, comprising: capturing a first pixel image at a primary sensor of at least a portion of the primary sensor's field of view; capturing a second pixel image at a first secondary sensor of at least a portion of the first secondary sensor's field of view; causing a first display screen to display the first pixel image; causing a second display screen to display the second pixel image; capturing a first set of depth values associated with a calibration object at the primary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; capturing a second set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; determining a first normal associated with a plane of the calibration object for the first set of depth values; determining a first on-plane point associated with the plane of the calibration object for the first set of depth values; determining a second normal associated with the plane of the calibration object for the second set of depth values; determining a second on-plane point associated with the plane of the calibration object for the second set of depth values; generating a first system of linear equations, in part, using a linear equation derived from the first normal and the second normal; solving for components of a rotation transform using the first system of linear equations; generating a second system of linear equations, in part, using a linear equation derived from the first on-plane point and the second on-plane point; and solving for components of a translation transform using the second system of linear equations, wherein the rotation transform and translation transform move a point from the perspective of the first secondary sensor to the perspective of the primary sensor. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the first display and the second display are the same display. ### Claim 9 (depends on 7) The computer-implemented method of claim 7, wherein the first on-plane point is an average of the depth values associated with the plane visible to the primary sensor. ### Claim 10 (depends on 7) The computer-implemented method of claim 7, wherein the calibration object is a planar board. ### Claim 11 (depends on 7) The computer-implemented method of claim 7, wherein the primary sensor's field of view and the first secondary sensor's field of view do not overlap. ### Claim 12 (depends on 7) The computer-implemented method of claim 7, the method further comprising: capturing a third set of depth values associated with the calibration object at a second secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; capturing a fourth set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; determining a third normal associated with a plane of the calibration object for the third set of depth values; determining a third on-plane point associated with the plane of the calibration object for the third set of depth values; determining a fourth normal associated with a plane of the calibration object for the fourth set of depth values; and determining a fourth on-plane point associated with the plane of the calibration object for the fourth set of depth values, wherein generating the first system of linear equations further comprises inserting a linear equation derived from the third normal and the fourth normal, and wherein generating the second system of linear equations further comprises inserting a linear equation derived from the third on-plane point and the fourth on-plane point. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein at least two of the plurality of sensors are vertically offset relative to one another by a bracket. ### Claim 14 A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method comprising: capturing a first set of depth values associated with a calibration object at a primary sensor when the calibration object is in both the primary sensor's field of view and a first secondary sensor's field of view; capturing a second set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; determining a first normal associated with a plane of the calibration object for the first set of depth values; determining a first on-plane point associated with the plane of the calibration object for the first set of depth values; determining a second normal associated with the plane of the calibration object for the second set of depth values; determining a second on-plane point associated with the plane of the calibration object for the second set of depth values; generating a first system of linear equations, in part, using a linear equation derived from the first normal and the second normal; solving for components of a rotation transform using the first system of linear equations; generating a second system of linear equations, in part, using a linear equation derived from the first on-plane point and the second on-plane point; and solving for components of a translation transform using the second system of linear equations, wherein the rotation transform and translation transform move a point from the perspective of the first secondary sensor to the perspective of the primary sensor. ### Claim 15 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the first on-plane point is an average of the depth values associated with the plane visible to the primary sensor. ### Claim 16 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the calibration object is a planar board. ### Claim 17 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the primary sensor's field of view and the first secondary sensor's field of view do not overlap. ### Claim 18 (depends on 14) The non-transitory computer-readable medium of claim 14, the method further comprising: capturing a third set of depth values associated with the calibration object at a second secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; capturing a fourth set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; determining a third normal associated with a plane of the calibration object for the third set of depth values; determining a third on-plane point associated with the plane of the calibration object for the third set of depth values; determining a fourth normal associated with a plane of the calibration object for the fourth set of depth values; and determining a fourth on-plane point associated with the plane of the calibration object for the fourth set of depth values, wherein generating the first system of linear equations further comprises inserting a linear equation derived from the third normal and the fourth normal, and wherein generating the second system of linear equations further comprises inserting a linear equation derived from the third on-plane point and the fourth on-plane point. ### Claim 19 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein at least two of the plurality of sensors are vertically offset relative to one another by a bracket. ### Claim 20 (depends on 1) The display structure of claim 1, wherein the display screen comprises multiple screens.