# US 10,484,530 B2 · Sensor based component activation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,530 B2 - Document ID: US10484530B2 - Status: Granted - Kind: B2 - Application number: 15/805,886 - Filed: 2017-11-07 - Published: 2019-05-09 - Granted: 2019-11-19 - Earliest claimed: 2017-11-07 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Tim Wantland - Web page: https://patentgrep.com/patent/US10484530B2 ## CPC - H04M 1/72454: according to context-related or environment-related conditions - G06F 1/1626: {with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]} - G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer} - G06F 1/3231: Monitoring the presence, absence or movement of users - G06F 1/3265: {Power saving in display device} - G06F 1/3287: by switching off individual functional units in the computer system - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/09: Supervised learning - G06N 3/098: Distributed learning, e.g. federated learning - G06F 2200/1637: Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer - G06N 3/084: Backpropagation, e.g. using gradient descent - H04M 2250/12: including a sensor for measuring a physical value, e.g. temperature or motion - H04M 2250/22: including a touch pad, a touch sensor or a touch detector - H04M 2250/52: including functional features of a camera - Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE ## Abstract Provided are methods, systems, and devices for activating components of a computing device based on a mode associated with the state of objects including a computing device. Sensor data based sensor outputs from one or more sensors that detect one or more objects can be received. Based in part on the sensor data, the state of the one or more objects can be determined. The state of the one or more objects can include one or more spatial relations of the one or more objects with respect to a computing system. When the state of the one or more objects satisfies one or more operational criteria, a mode associated with activating a component of the computing system can be activated. The component can include a microphone component, a camera component, or a text input component. The component can be associated with performing including input detection. ## Claims ### Claim 1 A computer-implemented method of mode determination, the method comprising: receiving, by a computing system comprising one or more computing devices, object detection data based in part on one or more sensor outputs from a plurality of sensors that detect one or more portions of a user of the computing system; receiving, by the computing system, motion data based in part on one or more sensor outputs from the plurality of sensors, wherein the motion data describes a motion of the computing system relative to the user of the computing system; determining, by the computing system, based in part on the object detection data, a state of the one or more portions of the user of the computing system comprising one or more spatial relations of the one or more portions of the user of the computing system with respect to the computing system; determining, by the computing system, based in part on the object detection data and the motion data, when the state of the one or more portions of the user relative to the computing system matches one or more falsing profiles associated with a state of the one or more portions of the user relative to the computing system that does not activate the one or more components, wherein the satisfying the one or more operational criteria comprises the state of the one or more portions of the user relative to the computing system not matching any of the one or more falsing profiles; and responsive to the state of the one or more portions of the user of the computing system satisfying one or more operational criteria, activating, by the computing system, an input mode of a plurality of input modes associated with activating one or more components of the computing system, the one or more components comprising one or more microphone components, one or more camera components, or one or more text input components, wherein the one or more components are associated with performing one or more operations comprising detection of one or more inputs. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, based in part on the one or more spatial relations of the one or more portions of the user, a distance or an angle between the computing system and the one or more portions of the user; and determining, by the computing system, when the distance or the angle between the computing system and the one or more portions of the user is within a distance range or angle range respectively, wherein the satisfying the one or more operational criteria comprises the determining that the distance or the angle between the computing system and the one or more portions of the user is within the distance range or the angle range respectively. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, based in part on the object detection data, when the one or more portions of the user comprise a face; determining, by the computing system, a distance between the face and the computing system; and determining, by the computing system, when the distance between the computing system and the face is within a distance range, wherein the satisfying the one or more operational criteria comprises the determining that the distance between the computing system and the face is within the distance range. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, further comprising: determining, by the computing system, based in part on the object detection data, a position of at least one eye of the face with respect to the computing system; and determining, by the computing system, based in part on the position of the at least one eye of the face with respect to the computing system, when the computing system is being gazed at for a predetermined period of time, wherein the satisfying the one or more operational criteria comprises the determining that the computing system is being gazed at for the predetermined period of time. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, based in part on the motion data, an acceleration of the computing system in one or more directions relative to the one or more portions of the user; and determining, by the computing system, when the acceleration of the computing system in a predetermined direction of the one or more directions exceeds an acceleration threshold, wherein the satisfying the one or more operational criteria comprises the acceleration of the computing system in the predetermined direction exceeding the acceleration threshold. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, based in part on the motion data, an acceleration and a duration of the acceleration of the computing system; and determining, by the computing system, based in part on the motion data, when the computing system undergoes acceleration within a predetermined acceleration range for a predetermined duration, wherein the satisfying the one or more operational criteria comprises the computing system undergoing acceleration within the predetermined acceleration range for the predetermined duration. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, based in part on the object detection data, that the computing system is being held in two hands when two of the one or more portions of the user are a predetermined distance range apart and in contact with one or more tactile sensors of the computing system, wherein the satisfying the one or more operational criteria comprises the determining that the computing system is being held in two hands. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, based in part on the object detection data, that the computing system is being held in two hands when two of the one or more portions of the user are within a predetermined distance and a field of view associated with a display output component of the computing system, wherein the satisfying the one or more operational criteria comprises the determining that the computing system is being held in two hands. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, based in part on the object detection data, that the computing system is being held in two hands when at least two of the one or more portions of the user are detected to be in contact with two or more portions of one or more tactile sensors behind a front portion of a display output component of the computing system and two of the one or more portions of the user are detected within a predetermined distance in front of the front portion of the display output component, wherein the satisfying the one or more operational criteria comprises the determining that the computing system is being held in two hands. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, when the one or more inputs are not received within a predetermined period of time after the activating the one or more components; generating, by the computing system, a falsing profile based in part on the object detection data and the motion data received within a falsing time interval comprising a time when the one or more components are activated and the one or more inputs are not received; and adding, by the computing system, the falsing profile to the one or more falsing profiles. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the one or more sensors comprise one or more image sensors, one or more audio sensors, one or more tactile sensors, one or more gyroscopic sensors, one or more accelerometers, one or more electromagnetic sensors, one or more thermal sensors, or one or more radar devices. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein determining, by the computing system, based in part on the object detection data and the motion data, the state of the one or more portions of the user, the state of the one or more portions of the user comprising one or more spatial relations of the one or more portions of the user with respect to the computing comprises: inputting, by the computing system, the object detection data and the motion data to a machine-learned mode determination model, wherein the machine-learned mode determination model has been trained to receive object detection data and motion data and, in response to receipt of the object detection data and motion data, output one or more predicted modes for operating the computing system; and receiving, by the computing system as an output of the machine-learned mode determination model, one or more predicted modes for operating the computing system. ### Claim 13 A mode determination system, comprising: one or more processors; a machine-learned mode determination model, wherein the machine-learned mode determination model has been trained to receive object detection data and, in response to receipt of the object detection data, output mode predictions for a computing device; and at least one tangible, non-transitory computer readable medium that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising: obtaining sensor data from a plurality of sensors of a computing device, wherein the sensor comprises at least object detection data and motion data, the object detection data associated with the state of one or more portions of a user of a computing device, wherein the motion data describes a motion of the computing device relative to the user of the computing device; inputting the object detection data into the machine-learned mode determination model, wherein the machine-learned mode determination model determines, based in part on the object detection data and the motion data, when the state of the one or more portions of the user relative to the computing device matches one or more falsing profiles associated with a state of the one or more portions of the user relative to the computing device that does not activate the one or more components, wherein the satisfying the one or more operational criteria comprises the state of the one or more portions of the user relative to the computing device not matching any of the one or more falsing profiles; and receiving, as output of the machine-learned mode determination model, a mode prediction for the computing device. ### Claim 14 (depends on 13) The mode determination system of claim 13, wherein the operations further comprise activating a component of the computing device that correlates with the mode prediction. ### Claim 15 (depends on 13) The mode determination system of claim 13, wherein the machine-learned mode determination model comprises a recurrent neural network. ### Claim 16 A computing device that automatically initiates an input mode, comprising: a plurality of sensors; at least one processor; and at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising: receiving object detection data based in part on one or more sensor outputs from the plurality of sensors that detect one or more portions of a user of the computing device; receiving motion data based in part on one or more sensor outputs from the plurality of sensors, the motion data describing a motion of the computing device relative to the user of the computing device; determining, based in part on the object detection data, a state of the one or more portions of the user of the computing device, the state of the one or more portions of the user of the computing device comprising one or more spatial relations of the one or more portions of the user of the computing device with respect to the computing device; determining, based in part on the object detection data and the motion data, when the state of the one or more portions of the user relative to the computing system matches one or more falsing profiles associated with a state of the one or more portions of the user relative to the computing system that does not activate the one or more components, wherein the satisfying the one or more operational criteria comprises the state of the one or more portions of the user relative to the computing system not matching any of the one or more falsing profiles; responsive to the state of the one or more portions of the user of the computing device satisfying one or more operational criteria, activating an input mode of a plurality of input modes associated with activating one or more components of the computing device, the one or more components comprising one or more microphone components, one or more camera components, or one or more text input components, wherein the one or more components are associated with performing one or more operations comprising detection of one or more inputs. ### Claim 17 (depends on 16) The computing device of claim 16, wherein the plurality of sensors include one or more accelerometers, one or more radar sensors, one or more tactile sensors, or one or more image sensors. ### Claim 18 (depends on 16) The computing device of claim 16, further comprising: determining, based in part on the one or more spatial relations of the one or more portions of the user, a distance or an angle between the computing device and the one or more portions of the user; and determining when the distance or the angle between the computing device and the one or more portions of the user is within a distance range or angle range respectively, wherein the satisfying the one or more operational criteria comprises the determining that the distance or the angle between the computing device and the one or more portions of the user is within the distance range or the angle range respectively. ### Claim 19 (depends on 16) The computing device of claim 16, further comprising: determining, based in part on the object detection data, when the one or more portions of the user comprise a face; determining a distance between the face and the computing device; and determining when the distance between the computing device and the face is within a distance range, wherein the satisfying the one or more operational criteria comprises the determining that the distance between the computing device and the face is within the distance range. ### Claim 20 (depends on 16) The computing device of claim 16, further comprising: determining when the one or more inputs are not received within a predetermined period of time after the activating the one or more components; generating a falsing profile based in part on the object detection data and the motion data received within a falsing time interval comprising a time when the one or more components are activated and the one or more inputs are not received; and adding the falsing profile to the one or more falsing profiles.