# US 2019/0356908 A1 · Interactive imaging systems and methods for motion control by users > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0356908 A1 - Document ID: US20190356908A1 - Status: Published application - Kind: A1 - Application number: 16/531,219 - Filed: 2019-08-05 - Published: 2019-11-21 - Earliest claimed: 2006-12-19 - Assignees: Cuesta Technology Holdings, LLC - Applicants: Cuesta Technology Holdings, LLC - Inventors: Matthew FLAGG; Greg ROBERTS - Web page: https://patentgrep.com/patent/US20190356908A1 ## CPC - H04N 17/002: {for television cameras} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06T 7/11: Region-based segmentation - G06T 7/194: involving foreground-background segmentation - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - H04N 23/11: for generating image signals from visible and infrared light wavelengths - G06T 2207/10012: Stereo images - G06T 2207/10048: Infrared image - G06T 2207/10152: Varying illumination - G06T 2207/30196: Human being; Person - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In various embodiments, the present invention provides a system and associated methods of calibration and use for an interactive imaging environment based on the optimization of parameters used in various segmentation algorithm techniques. These methods address the challenge of automatically calibrating an interactive imaging system, so that it is capable of aligning human body motion, or the like, to a visual display. As such the present invention provides a system and method of automatically and rapidly aligning the motion of an object to a visual display. ## Claims ### Claim 1 A method of calibrating an interactive imaging system, the method comprising the steps of: performing a geometric calibration process to create warp parameters to compensate for a parallax effect by mapping a sensed infrared view to a projection image; and performing a photometric calibration process by matching a shadow image with the projection image. ### Claim 2 (depends on 1) The method of claim 1, wherein mapping the sensed infrared view to the projection image includes: mapping, according to a first mapping process, the sensed infrared view to a sensed visible-light view; mapping, according to a second mapping process, the sensed visible-light view to the projection image; and performing a homography process by combining the first and second mapping processes. ### Claim 3 (depends on 2) The method of claim 2, wherein the homography process is a two-dimensional perspective transformation that maps pixels on a first plane defined by the sensed infrared view to pixels on a second plane defined by the projection image, the transformation made through an intermediate plane defined by the sensed visible-light view. ### Claim 4 (depends on 3) The method of claim 3, further comprising the steps of: obtaining the sensed infrared view by detecting at least three non-linear reference points lying in a plane; and obtaining the sensed visible-light view by detecting the at least three non-linear reference points and detecting corners of the projection image. ### Claim 5 (depends on 1) The method of claim 1, wherein the steps of performing the geometric calibration and performing the photometric calibration includes initializing and optimizing setup parameters used in a segmentation algorithm for execution after calibration is complete. ### Claim 6 (depends on 5) The method of claim 5, wherein the segmentation algorithm includes a segmentation process of splitting up the projection image into segments, wherein each segment holds properties distinct from areas adjacent to the respective segment. ### Claim 7 (depends on 5) The method of claim 5, wherein the segmentation algorithm includes tracking, in real time, at least a head and/or pair of hands of a user creating the shadow image. ### Claim 8 (depends on 7) The method of claim 7, wherein the tracking in real time includes performing the tracking within 1/60th of a second. ### Claim 9 (depends on 5) The method of claim 5, wherein the segmentation algorithm includes background subtraction and stereo disparity-based foreground detection. ### Claim 10 (depends on 9) The method of claim 9, wherein the background subtraction includes the steps of: setting the warp parameters; and tuning a threshold variable t, a number n of median filter operations, a size m of median filter kernels, and an exposure e of a camera configured to sense a visible-light image of a user casting the shadow image; wherein tuning the variables t, n, m, and e optimizes an objective function. ### Claim 11 (depends on 10) The method of claim 10, wherein optimizing the objective function includes evaluating a difference between the segmentation algorithm computed with given parameters and the segmentation algorithm based on an expected segmentation. ### Claim 12 (depends on 1) The method of claim 1, wherein the step of performing the photometric calibration includes the steps of: allowing a user to stand between an image generator and a display medium, wherein the image generator is configured to project the projection image and the display medium configured to reflect the projection image; and allowing the user to move into a position such that the shadow image cast by the user mimics a line-up silhouette of the projection image. ### Claim 13 (depends on 12) The method of claim 12, wherein the step of performing the photometric calibration further includes the steps of: creating a segmented silhouette from the shadow image; calculating a score indicative of the differences between coordinates in the line-up silhouette and coordinates in the segmented silhouette; optimizing setup parameters that result in a minimization of the calculated score; and indicating the interactive imaging system as being calibrated when the calculated score reaches a minimum score representing a substantial match. ### Claim 14 An interactive imaging system, comprising: a display medium configured to display a projection image having a silhouette feature; an infrared image sensor having a first sensing view obtained from a first perspective for sensing infrared signals of the projection image; and a control unit configured to: perform a geometric calibration process to create warp parameters to compensate for a parallax effect by mapping the sensed infrared signals obtained from the first perspective to the projection image; and perform a photometric calibration process by matching a shadow image with the projection image. ### Claim 15 (depends on 14) The interactive imaging system of claim 14, further comprising a visible-light image sensor having a second sensing view obtained from a second perspective for sensing visible-light signals of the projection image. ### Claim 16 (depends on 15) The interactive imaging system of claim 15, wherein the control unit is configured to map the sensed infrared signals to the projection image by: mapping, according to a first mapping process, the sensed infrared signals to the sensed visible-light signals; mapping, according to a second mapping process, the sensed visible-light signals to the projection image; and performing a homography process by combining the first and second mapping processes; wherein the homography process is a two-dimensional perspective transformation that maps pixels on a first plane defined by the sensed infrared signals to pixels on a second plane defined by the projection image, the transformation made through an intermediate plane defined by the sensed visible-light signals. ### Claim 17 (depends on 15) The interactive imaging system of claim 15, wherein the infrared image sensor is configured to obtain the sensed infrared signals by detecting at least three non-linear reference points lying in a plane, and wherein the visible-light image sensor is configured to obtain the sensed visible-light signals by detecting the at least three non-linear reference points and detecting corners of the projection image on the display medium. ### Claim 18 (depends on 14) The interactive imaging system of claim 14, wherein the control unit is further configured to optimize setup parameters of a segmentation algorithm for splitting up the projection image into segments, and wherein each segment holds properties distinct from areas adjacent to the respective segment. ### Claim 19 (depends on 18) The interactive imaging system of claim 18, wherein the segmentation algorithm includes background subtraction and stereo disparity-based foreground detection. ### Claim 20 (depends on 14) The interactive imaging system of claim 14, wherein the control unit is configured to perform the photometric calibration by: allowing a user to stand between an image generator and a display medium, wherein the image generator is configured to project the projection image; and allowing the user to move into a position such that a shadow image cast by the user mimics the silhouette feature of the projection image.