# US 9,436,277 B2 · System and method for producing computer control signals from breath attributes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,436,277 B2 - Document ID: US9436277B2 - Status: Granted - Kind: B2 - Application number: 14/257,393 - Filed: 2014-04-21 - Published: 2015-10-22 - Granted: 2016-09-06 - Earliest claimed: 2014-04-21 - Assignees: Xerox Corporation - Applicants: Xerox Corporation - Inventors: Michael R. Furst; Edgar A. Bernal; Robert P. Loce; Lalit K. Mestha - Web page: https://patentgrep.com/patent/US9436277B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens} - A61B 5/08: Measuring devices for evaluating the respiratory organs - A61B 5/0816: {Measuring devices for examining respiratory frequency (measuring frequency of electric signals G01R23/00)} - A61B 5/087: Measuring breath flow - A61B 5/1135: {by monitoring thoracic expansion} - A61B 5/7485: {Automatic selection of region of interest} - G06F 3/005: {Input arrangements through a video camera} - G06F 3/012: {Head tracking input arrangements} - A61B 2560/0223: of calibration, e.g. protocols for calibrating sensors - A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence) - G16H 30/40: for processing medical images, e.g. editing - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract A method for computing output using a non-contact (invisible) input signal includes acquiring depth data of a scene captured by a depth-capable sensor. The method includes generating a temporal series of depth maps corresponding to the depth data. The method includes generating at least one volumetric attribute from the depth data. The method includes generating an output based on the volumetric attribute to control actions. ## Claims ### Claim 1 A system for computing output using a non-contact input signal, the system comprising: a computer programmed to perform a method for generating an input control signal and including the operations of: acquiring depth data of a scene captured by a depth-capable sensor; generating a temporal series of depth maps corresponding to the depth data; generating at least one volumetric attribute based on the depth data determined from the series of depth maps, the generating including: for each depth map, generating a binary image of a subject detected in the each depth map, segmenting the binary image into regions, comparing pixels corresponding to a select region of interest across the temporal series of depth maps to determine a change in a shape of the region, associating the change in the shape with a volumetric attribute; determining one of an optimal pose of the subject and an optimal orientation of the subject using a breath analyzer in communication with the computer; providing the one of the pose and orientation to an output device as an output, and providing a pose recommendation feedback for improving a quality of the depth data; and generating an input control signal based on the volumetric attribute to enable the subject to interact with an output device. ### Claim 2 (depends on 1) The system of claim 1, wherein the computer is further operative to localize the subject by at least one of detecting and tracking the subject across the series of depth maps. ### Claim 3 (depends on 1) The system of claim 1, wherein the depth-capable sensor includes: a first image capture device located at a first position and capturing a first view of a scene; and, a second image capture device located at a second position separated a predetermined distance from the first image capture device and capturing a second view of the scene offset a predetermined degree from the first view of the scene. ### Claim 4 (depends on 1) The system of claim 1, wherein the depth-capable sensor is selected from a group consisting of: an active stereo device comprising a structured illumination system operative to project patterned illumination onto the subject, and an image acquisition device operative to image a reflected patterned illumination; a time of flight system measuring a time of flight of a light signal between a light source and the subject for each point of an image using a laser or light pulse; and, a combination of the above. ### Claim 5 (depends on 1) The system of claim 1 wherein the computer is further operative to generate information regarding a location of pixels corresponding to a region of interest (“ROI”) within an image of the scene. ### Claim 6 (depends on 5) The system of claim 5, wherein the ROI includes at least one of a chest and abdominal area. ### Claim 7 (depends on 1) The system of claim 1, wherein the volumetric attribute is selected from a group consisting of: amplitude of a change in volume within a predetermined time interval; a rate of the change of a volume; and, a combination of the amplitude and the rate. ### Claim 8 (depends on 1) The system of claim 1, wherein the computer further includes the breath analyzer and is operative to: determine an orientation of a face of the subject and estimate a direction of inhale and exhale air flow created by breathing. ### Claim 9 (depends on 1) The system of claim 1, wherein the output includes a control signal for controlling an associated device. ### Claim 10 (depends on 1) The system of claim 1, wherein the output is a characteristic breathing pattern. ### Claim 11 (depends on 10) The system of claim 10, wherein the output device is a gaming device and the breathing pattern is introduced in a gaming scene of the subject. ### Claim 12 A method for computing output using a non-contact input signal, the method including: acquiring depth data of a scene captured by a depth-capable sensor; generating a temporal series of depth maps corresponding to the depth data; generating at least one volumetric attribute based on the depth data determined from the series of depth maps, the generating including: for each depth map, generating a binary image of a subject detected in the each depth map, segmenting the binary image into regions, comparing pixels corresponding to a select region of interest across the temporal series of depth maps to determine a change in a shape of the region, associating the change in the shape with a volumetric attribute; determining one of an optimal pose of the subject and an optimal orientation of the subject using a breath analyzer in communication with the computer; providing the one of the pose and orientation to an output device as an output, and providing a pose recommendation feedback for improving a quality of the depth data; and generating an input control signal based on the volumetric attribute to enable the subject to interact with an output device. ### Claim 13 (depends on 12) The method of claim 12, wherein the depth-capable sensor includes: a first image capture device located at a first position and capturing a first view of a scene; and, a second image capture device located at a second position separated a predetermined distance from the first image capture device and capturing a second view of the scene offset a predetermined degree from the first view of the scene. ### Claim 14 (depends on 12) The method of claim 12, wherein the depth-capable sensor includes: an active stereo device comprising a structured illumination system operative to project patterned illumination onto the subject, and an image acquisition device operative to image a reflected patterned illumination. ### Claim 15 (depends on 12) The method claim 12, further comprising: determining pixels corresponding to a region of interest (“ROI”) being monitored on a subject by the depth-capable sensor. ### Claim 16 (depends on 15) The method of claim 15, wherein the ROI includes at least one of a chest and abdominal area. ### Claim 17 (depends on 12) The method of claim 12, wherein the volumetric attribute is selected from a group consisting of: amplitude of a change in volume within a predetermined time interval; a rate of the change of a volume; and, a combination of the amplitude and the rate. ### Claim 18 (depends on 12) The method of claim 12, further comprising: determining an orientation of a face of the subject and estimating a direction of inhale and exhale air flow created by breathing activity of the subject. ### Claim 19 (depends on 12) The method of claim 12, wherein the output includes a control signal for controlling an associated device. ### Claim 20 (depends on 19) The method of claim 19, wherein the output includes a breathing pattern and the associated device is a gaming device for determining vital capacities of the subject. ### Claim 21 (depends on 12) The method of claim 12, further comprising: calibrating the output device by performing a breathing cycle selected from a group consisting: at least one tidal breathing cycle; a forced inspiration cycle; a forced expiration cycle; and, a combination of the above. ### Claim 22 (depends on 21) The method of claim 21, further comprising: computing maximum vital capacities of the subject using the breathing cycle.