# US 9,517,776 B2 · Systems, methods, and apparatus for controlling devices based on a detected gaze > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,517,776 B2 - Document ID: US9517776B2 - Status: Granted - Kind: B2 - Application number: 13/977,516 - Filed: 2011-12-29 - Published: 2014-11-27 - Granted: 2016-12-13 - Earliest claimed: 2011-12-29 - Assignees: Intel Corporation - Applicants: David L. Graumann - Inventors: David L. Graumann - Web page: https://patentgrep.com/patent/US9517776B2 ## CPC - B60W 50/10: Interpretation of driver requests or demands - B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor - B60W 50/08: Interaction between the driver and the control system - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - G06V 40/19: Sensors therefor - B60K 35/22: Display screens - B60K 35/265: {Voice} - B60K 37/00: Dashboards (as road-vehicle superstructure sub-units B62D25/14) - B60K 2360/149: Instrument input by detecting viewing direction not otherwise provided for - B60K 2360/21: using cameras - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2540/00: Input parameters relating to occupants - B60W 2540/21: Voice - B60W 2540/22: Psychological state; Stress level or workload - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Certain embodiments of the invention may include systems, methods, and apparatus for controlling devices based on a detected gaze. According to an example embodiment of the invention, a method is provided for executing computer executable instructions by one or more processors for controlling one or more devices attached to a vehicle based at least in part on a detected direction of observation of an occupant of the vehicle, the method further includes receiving one or more images from at least one camera attached to the vehicle; locating, from the one or more images, one or more body features associated with the occupant; determining a direction of gaze from the one or more located body features; and controlling the one or more devices based at least in part on the direction of gaze. ## Claims ### Claim 1 A method comprising executing computer executable instructions by one or more processors for controlling at least a first device and a second device attached to a vehicle based at least in part on a detected direction of observation of an occupant of the vehicle, the method further comprising: receiving one or more images from at least one camera attached to the vehicle, the vehicle including an object comprising a first portion associated with the first device and a second portion associated with the second device; locating, from the one or more images, one or more body features associated with the occupant; determining a direction of gaze towards the object using the one or more located body features; and controlling a first output of the first device associated with the first portion and controlling a second output of the second device associated with the second portion, based on the determined direction of gaze being towards the first portion, wherein controlling the second output of the second device comprises altering the second output. ### Claim 2 (depends on 1) The method of claim 1, wherein the locating of the one or more body features comprises locating at least a head associated with the occupant. ### Claim 3 (depends on 1) The method of claim 1, wherein the locating of the one or more body features comprises locating one or more eyes associated with the occupant. ### Claim 4 (depends on 1) The method of claim 1, further comprising determining a gesture of the occupant from the one or more body features and controlling at least one of the first device and the second device based at least in part on the gesture. ### Claim 5 (depends on 4) The method of claim 4, wherein the gesture comprises one or more of a hand movement or a voice command. ### Claim 6 (depends on 1) The method of claim 1, wherein altering the second output comprises one or more of adjusting a brightness or contrast of at least a portion of a display, actuating a motor, adjusting a control setting, or prompting the occupant with audible or visual information. ### Claim 7 (depends on 1) The method of claim 1, wherein determining the direction of gaze comprises tracking the occupant's head attitude duration and stability. ### Claim 8 (depends on 1) The method of claim 1, wherein the first device and the second device comprise an integrated device. ### Claim 9 A vehicle comprising: a first device attached to the vehicle; a second device attached to the vehicle; at least one camera attached to the vehicle, the vehicle including an object comprising a first portion associated with the first device and a second portion associated with the second device; at least one memory for storing data and computer-executable instructions; and one or more processors configured to access the at least one memory and further configured to execute computer executable instructions for: receiving one or more images from at least one camera attached to the vehicle; locating, from the one or more images, one or more body features associated with an occupant of the vehicle; determining a direction of gaze towards the object using the one or more located body features; and controlling a first output of the first device associated with the first portion and controlling a second output of the second device associated with the second portion, based on the determined direction of gaze being towards the first portion, wherein controlling the second output of the second device comprises altering the second output. ### Claim 10 (depends on 9) The vehicle of claim 9, wherein the locating of the one or more body features comprises locating at least a head associated with the occupant. ### Claim 11 (depends on 9) The vehicle of claim 9, wherein the locating of the one or more body features comprises locating one or more eyes associated with the occupant. ### Claim 12 (depends on 9) The vehicle of claim 9, wherein the one or more processors are further configured for determining a gesture of the occupant from the one or more body features and controlling at least one of the first device and the second device based at least in part on the gesture. ### Claim 13 (depends on 12) The vehicle of claim 12, wherein the gesture comprises one or more of a hand movement or a voice command. ### Claim 14 (depends on 9) The vehicle of claim 9, wherein altering the second output comprises one or more of adjusting a brightness or contrast of at least a portion of a display, actuating a motor, adjusting a control setting, or prompting the occupant with audible or visual information. ### Claim 15 (depends on 9) The vehicle of claim 9, wherein determining the direction of gaze comprises tracking the occupant's head attitude duration and stability. ### Claim 16 (depends on 9) The vehicle of claim 9, wherein the at least one of the first device and the second device comprises one or more of a video screen, a mirror with an integrated display, a stereo system, a sun roof, a car window, a lock, a car seat, a climate control system, or a dashboard console. ### Claim 17 (depends on 9) The vehicle of claim 9, wherein the first device and the second device comprise an integrated device. ### Claim 18 An apparatus comprising: at least one memory for storing data and computer-executable instructions; and one or more processors configured to access the at least one memory and further configured to execute computer executable instructions for: receiving one or more images from at least one camera attached to the vehicle, the vehicle including an object comprising a first portion associated with a first device and a second portion associated with a second device; locating, from the one or more images, one or more body features associated with an occupant of the vehicle; determining a direction of gaze towards the object using the one or more located body features; and controlling a first output of the first device associated with the first portion and controlling a second output of the second device associated with the second portion, based on the determined direction of gaze being towards the first portion, wherein controlling the second output of the second device comprises altering the second output. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein the locating of the one or more body features comprises locating at least a head associated with the occupant. ### Claim 20 (depends on 18) The apparatus of claim 18, wherein the locating of the one or more body features comprises locating one or more eyes associated with the occupant. ### Claim 21 (depends on 18) The apparatus of claim 18, wherein the one or more processors are further configured for determining a gesture of the occupant from the one or more body features and controlling at least one of the first device and the second device based at least in part on the gesture. ### Claim 22 (depends on 21) The apparatus of claim 21, wherein the gesture comprises one or more of a hand movement or a voice command. ### Claim 23 (depends on 18) The apparatus of claim 18, wherein altering the second output comprises one or more of adjusting a brightness or contrast of at least a portion of a display, actuating a motor, adjusting a control setting, or prompting the occupant with audible or visual information. ### Claim 24 (depends on 18) The apparatus of claim 18, wherein determining the direction of gaze comprises tracking the occupant's head attitude duration and stability. ### Claim 25 (depends on 18) The apparatus of claim 18, wherein at least one of the first device and the second device comprises one or more of a video screen, a mirror with an integrated display, a stereo system, a sun roof, a car window, a lock, a car seat, a climate control system, or a dashboard console. ### Claim 26 (depends on 18) The apparatus of claim 18, wherein the first device and the second device comprise an integrated device. ### Claim 27 A computer program product, comprising a non-transitory computer-readable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for controlling at least a first device and a second device attached to a vehicle based at least in part on a detected direction of observation of an occupant of the vehicle, the method further comprising: receiving one or more images from at least one camera attached to the vehicle, the vehicle including an object comprising a first portion associated with the first device and a second portion associated with the second device; locating, from the one or more images, one or more body features associated with the occupant; determining a direction of gaze towards the object using the one or more located body features; and controlling a first output of the first device associated with the first portion and controlling a second output of the second device associated with the second portion, based on the determined direction of gaze being towards the first portion, wherein controlling the second output of the second device comprises altering the second output. ### Claim 28 (depends on 27) The computer program product of claim 27, wherein the locating of the one or more body features comprises locating at least a head associated with the occupant. ### Claim 29 (depends on 27) The computer program product of claim 27, wherein the locating of the one or more body features comprises locating one or more eyes associated with the occupant. ### Claim 30 (depends on 27) The computer program product of claim 27, further comprising determining a gesture of the occupant from the one or more body features and controlling at least one of the first device and the second device based at least in part on the gesture. ### Claim 31 (depends on 30) The computer program product of claim 30, wherein the gesture comprises one or more of a hand movement or a voice command. ### Claim 32 (depends on 27) The computer program product of claim 27, wherein altering the second output comprises one or more of adjusting a brightness or contrast of at least a portion of a display, actuating a motor, adjusting a control setting, or prompting the occupant with audible or visual information. ### Claim 33 (depends on 27) The computer program product of claim 27, wherein determining the direction of gaze comprises tracking the occupant's head attitude duration and stability. ### Claim 34 (depends on 27) The computer program product of claim 27, wherein the first device and the second device comprise an integrated device.