# US 12,000,609 B2 · Detecting presence and estimating thermal comfort of one or more human occupants in a built space in real-time using one or more thermographic cameras and one or more RGB-D sensors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,000,609 B2 - Document ID: US12000609B2 - Status: Granted - Kind: B2 - Application number: 17/422,142 - Filed: 2020-01-09 - Published: 2022-03-24 - Granted: 2024-06-04 - Earliest claimed: 2019-01-10 - Assignees: The Regents of the University of Michigan - Applicants: The Regents of the University of Michigan - Inventors: Da Li; Carol C. Menassa; Vineet Kamat - Web page: https://patentgrep.com/patent/US12000609B2 ## CPC - F24F 11/63: Electronic processing - B60H 1/00742: {by detection of the vehicle occupants' presence; by detection of conditions relating to the body of occupants, e.g. using radiant heat detectors} - G01J 5/0025: {Living bodies (ear thermometers G01J5/0011; detecting, measuring or recording for diagnostic purposes A61B5/00)} - G06T 7/33: using feature-based methods - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - F24F 2120/00: Control inputs relating to users or occupants - F24F 2120/10: Occupancy - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10048: Infrared image - G06T 2207/20081: Training; Learning - G06T 2207/30196: Human being; Person - G06T 2207/30201: Face - G06T 2207/30232: Surveillance - F24F: AIR-CONDITIONING; AIR-HUMIDIFICATION; VENTILATION; USE OF AIR CURRENTS FOR SCREENING (removing dirt or fumes from areas where they are produced B08B15/00; vertical ducts for carrying away waste gases from buildings E04F17/02; tops for chimneys or ventilating shafts, terminals for flues F23L17/02) - B60H: ARRANGEMENTS OF HEATING, COOLING, VENTILATING OR OTHER AIR-TREATING DEVICES SPECIALLY ADAPTED FOR PASSENGER OR GOODS SPACES OF VEHICLES - G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method of making a thermal comfort estimation of one or more human occupants in a built space in real-time is presented. The method employs one or more thermographic cameras and one or more red, green, and blue depth (RGB-D) sensors. The method has several steps. In an example, the method involves capturing thermal images of the human occupant(s) via the thermographic camera(s) and capturing RGB-D images of the human occupant(s) via the RGB-D camera(s). The method further involves extracting facial skin temperatures of the human occupant(s) using the captured thermal images and using the captured RGB-D images. And the method involves estimating the thermal comfort of the human occupant(s) using the extracted facial skin temperatures. ## Claims ### Claim 1 A method of making a thermal comfort estimation of at least one human occupant in a built space in real-time with the use of at least one thermographic camera and at least one red, green, and blue depth (RGB-D) sensor, the method comprising: providing the at least one thermographic camera at the built space, and providing the at least one RGB-D sensor at the built space; capturing thermal images of the at least one human occupant in the built space via the at least one thermographic camera, and capturing RGB-D images of the at least one human occupant in the built space via the at least one RGB-D sensor; extracting facial skin temperatures of the at least one human occupant using the captured thermal images and the captured RGB-D images, wherein extracting facial skin temperatures involves registering the captured thermal images and captured RGB-D images; and estimating the thermal comfort of the at least one human occupant in the built space using the extracted facial skin temperature of the at least one human occupant; wherein capturing thermal images of the at least one human occupant in the built space involves capturing the thermal images from various angles of the at least one human occupant relative to the at least one thermographic camera, and capturing RGB-D images of the at least one human occupant in the built space involves capturing the RGB-D images from various angles of the at least one human occupant relative to the at least one RGB-D sensor. ### Claim 2 (depends on 1) The method as set forth in claim 1, further comprising controlling heating, ventilation, and air conditioning (HVAC) of the built space based on the estimated thermal comfort of the at least one human occupant. ### Claim 3 (depends on 2) The method as set forth in claim 2, wherein controlling the HVAC of the built space involves determining a thermal setpoint of the built space based on the estimated thermal comfort of the at least one human occupant. ### Claim 4 (depends on 1) The method as set forth in claim 1, wherein providing the at least one thermographic camera at the built space involves providing multiple thermographic cameras at the built space, and providing the at least one RGB-D sensor at the built space involves providing multiple RGB-D sensors at the built space, and wherein capturing thermal images of the at least one human occupant in the built space involves capturing thermal images of multiple human occupants in the built space via the multiple thermographic cameras, and capturing RGB-D images of the at least one human occupant in the built space involves capturing RGB-D images of multiple occupants in the built space via the multiple RGB-D sensors. ### Claim 5 (depends on 1) The method as set forth in claim 1, wherein providing the at least one thermographic camera at the built space involves providing multiple thermographic cameras at the built space, and providing the at least one RGB-D sensor at the built space involves providing multiple RGB-D sensors at the built space, wherein capturing thermal images of the at least one human occupant involves capturing the thermal images of multiple human occupants via the multiple thermographic cameras, capturing RGB-D images of the at least one human occupant involves capturing the RGB-D images of the multiple human occupants via the multiple RGB-D sensors, and wherein the method further comprises registering the multiple human occupants among the multiple thermographic cameras and among the multiple RGB-D sensors. ### Claim 6 (depends on 1) The method as set forth in claim 1, wherein capturing thermal images of the at least one human occupant in the built space involves capturing the thermal images from various distances of the at least one human occupant relative to the at least one thermographic camera, and capturing RGB-D images of the at least one human occupant in the built space involves capturing the RGB-D images from various distances of the at least one human occupant relative to the at least one RGB-D sensor. ### Claim 7 (depends on 1) The method as set forth in claim 1, wherein extracting facial skin temperatures comprises detecting the presence of the at least one human occupant in the built space via the at least one RGB-D sensor. ### Claim 8 (depends on 1) The method as set forth in claim 1, wherein registering the captured thermal images and captured RGB-D images involves locating coordinates in the captured RGB-D images onto the captured thermal images. ### Claim 9 (depends on 1) The method as set forth in claim 1, wherein estimating the thermal comfort of the at least one human occupant in the built space involves estimating a warmer thermal comfort preference, a cooler thermal comfort preference, or a neutral thermal comfort preference. ### Claim 10 (depends on 1) The method as set forth in claim 1, wherein extracting facial skin temperatures comprises using distance data provided from the at least one RGB-D sensor with temperature data provided from the at least one thermographic camera. ### Claim 11 (depends on 2) The method as set forth in claim 2, wherein the built space is within a transportation vehicle. ### Claim 12 A method of making a thermal comfort estimation of at least one human occupant in a built space in real-time with the use of at least one thermographic camera and at least one red, green, and blue depth (RGB-D) sensor, the method comprising: capturing thermal images of the at least one human occupant in the built space via the at least one thermographic camera from various distances and various angles of the at least one human occupant relative to the at least one thermographic camera, and capturing RGB-D images of the at least one human occupant in the built space via the at least one RGB-D sensor from various distances and various angles of the at least one human occupant relative to the at least one RGB-D sensor; extracting facial skin temperatures of the at least one human occupant using the captured thermal images and the captured RGB-D images, distance data provided from the at least one RGB-D sensor is used with temperature data provided from the at least one thermographic camera; and estimating the thermal comfort of the at least one human occupant in the built space using the extracted facial skin temperature of the at least one human occupant. ### Claim 13 (depends on 12) The method as set forth in claim 12, further comprising controlling heating, ventilation, and air conditioning (HVAC) of the built space based on the estimated thermal comfort of the at least one human occupant. ### Claim 14 (depends on 12) The method as set forth in claim 12, wherein capturing thermal images of the at least one human occupant in the built space involves capturing thermal images of multiple human occupants in the built space via multiple thermographic cameras, and capturing RGB-D images of the at least one human occupant in the built space involves capturing RGB-D images of multiple occupants in the built space via multiple RGB-D sensors. ### Claim 15 (depends on 12) The method as set forth in claim 12, wherein capturing thermal images of the at least one human occupant involves capturing the thermal images of multiple human occupants via multiple thermographic cameras, capturing RGB-D images of the at least one human occupant involves capturing the RGB-D images of the multiple human occupants via multiple RGB-D sensors, and wherein the method further comprises registering the multiple human occupants among the multiple thermographic cameras and among the multiple RGB-D sensors. ### Claim 16 (depends on 12) The method as set forth in claim 12, wherein extracting facial skin temperatures involves registering the captured thermal images and captured RGB-D images. ### Claim 17 A method of making a thermal comfort estimation of at least one human occupant in a built space in real-time with the use of at least one thermographic camera and at least one red, green, and blue depth (RGB-D) sensor, the method comprising: capturing thermal images of the at least one human occupant in the built space via the at least one thermographic camera, and capturing RGB-D images of the at least one human occupant in the built space via the at least one RGB-D sensor; extracting facial skin temperatures of the at least one human occupant using the captured thermal images and the captured RGB-D images, wherein extracting facial skin temperatures involves locating coordinates in the captured RGB-D images onto the captured thermal images; estimating the thermal comfort of the at least one human occupant in the built space using the extracted facial skin temperature of the at least one human occupant; and controlling heating, ventilation, and air conditioning (HVAC) of the built space based on the estimated thermal comfort of the at least one human occupant; wherein capturing thermal images of the at least one human occupant in the built space involves capturing the thermal images from various angles of the at least one human occupant relative to the at least one thermographic camera, and capturing RGB-D images of the at least one human occupant in the built space involves capturing the RGB-D images from various angles of the at least one human occupant relative to the at least one RGB-D sensor. ### Claim 18 (depends on 17) The method as set forth in claim 17, further comprising determining the total number of human occupants in the built space using the captured thermal images. ### Claim 19 (depends on 12) A non-transitory computer readable medium comprising a non-transient data storage device having stored thereon instructions that carry out the method of claim 12.