# US 12,118,790 B2 · Auto-capture of interesting moments by assistant systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,790 B2 - Document ID: US12118790B2 - Status: Granted - Kind: B2 - Application number: 17/394,096 - Filed: 2021-08-04 - Published: 2022-11-17 - Granted: 2024-10-15 - Earliest claimed: 2021-04-21 - Assignees: Meta Platforms, Inc. - Applicants: Meta Platforms, Inc. - Inventors: Emily Wang; Yilei Li; Amy Lawson Bearman; Alireza Dirafzoon; Ruchir Srivastava - Web page: https://patentgrep.com/patent/US12118790B2 ## CPC - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06V 20/47: {Detecting features for summarising video content} - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06N 20/20: Ensemble learning - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - H04N 23/60: Control of cameras or camera modules - H04N 23/61: based on recognised objects - H04N 23/63: by using electronic viewfinders - H04N 23/633: {for displaying additional information relating to control or operation of the camera} - H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image} - H04N 23/661: Transmitting camera control signals through networks, e.g. control via the Internet - H04N 23/6812: {based on additional sensors, e.g. acceleration sensors} - H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06V 20/44: {Event detection} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract In one embodiment, a method includes accessing from a client system associated with a first user sensor signals captured by sensors of the client system, wherein the client system comprises a plurality of sensors, and wherein the sensors signals are accessed from the sensors based on cascading model policies, wherein each cascading model policy utilizes one or more of a respective cost or relevance associated with each sensor, detecting a change in a context of the first user associated with an activity of the first user based on machine-learning models and the sensor signals, wherein the change in the context of the first user satisfies a trigger condition associated with the activity, and responsive to the detected change in the context of the first user automatically capturing visual data by cameras of the client system. ## Claims ### Claim 1 A method comprising, by a client system: accessing, from the client system associated with a first user, one or more sensor signals captured by one or more sensors of the client system, wherein the client system comprises a plurality of sensors, and wherein the one or more sensors signals are accessed from the one or more sensors based on one or more cascading model policies, wherein each cascading model policy utilizes one or more of a respective cost or relevance associated with each sensor; detecting, based on one or more machine-learning models and the one or more sensor signals, a change in a context of the first user associated with an activity of the first user, wherein the change in the context of the first user satisfies a trigger condition associated with the activity; automatically capturing, responsive to the detected change in the context of the first user, visual data by one or more cameras of the client system, receiving, at the client system, a user request to capture visual data associated with the activity; and determining the trigger condition based on the user request. ### Claim 2 (depends on 1) The method of claim 1, wherein the visual data comprises one or more moments of the activity, wherein each of the one or more moments is associated with a measure of interestingness greater than a threshold measure of interestingness. ### Claim 3 (depends on 2) The method of claim 2, further comprising: determining, based on the one or more machine-learning models and the one or more sensor signals, that the measure of interestingness with each of the one or more moments is higher than the threshold measure of interestingness. ### Claim 4 (depends on 3) The method of claim 3, wherein one or more of the one or more machine-learning models are trained based on a plurality of sensor signals corresponding to a plurality of moments of a plurality of activities, wherein each of the plurality of sensor signals is associated with a predetermined measure of interestingness. ### Claim 5 (depends on 1) The method of claim 1, wherein the visual data is stored as one or more media data files, wherein each of the one or more media data files comprises one or more of an image or a video. ### Claim 6 (depends on 1) The method of claim 1, further comprising: detecting, based on the one or more machine-learning models and the one or more sensor signals, an end to the activity; generating, based on the visual data captured during the activity, a highlights video based on a summarization of the visual data, wherein the summarization is based on a measure of interestingness of the visual data; and presenting, at the client system, the highlights video to the first user. ### Claim 7 (depends on 1) The method of claim 1, wherein the one or more sensor signals comprise one or more of an inertial measurement unit (IMU) signal, an audio signal, a GPS signal, an electromyography (EMG) signal, or a visual signal. ### Claim 8 (depends on 1) The method of claim 1, wherein each cascading model policy utilizes a respective cost associated with each sensor, and wherein each cascading model policy specifies accessing a first sensor signal from a first sensor associated with a first cost before accessing a second sensor signal from a second sensor associated with a second cost if the first cost is lower than the second cost. ### Claim 9 (depends on 1) The method of claim 1, wherein detecting the change in the context of the first user associated with the activity of the first user comprises: calculating, based on the one or more machine-learning models and a first sensor signal of a first sensor of the plurality of sensors, a first confidence score associated with the change in the context of the first user associated with the activity of the first user; determining whether the first confidence score is greater than a threshold confidence score; and based on the determining of whether the first confidence score is greater than the threshold score: if the first confidence score is greater than the threshold score, confirming the change in the context of the first user associated with the activity of the first user; else: accessing, from the client system associated with the first user, a second sensor signal of a second sensor of the plurality of sensors, wherein the second sensor signal is accessed based on the one or more cascading model policies; and calculating, based on the one or more machine-learning models and the second sensor signal, a second confidence score associated with the change in the context of the first user associated with the activity of the first user, wherein if the second confidence score is greater than the threshold score, confirming the change in the context of the first user associated with the activity of the first user. ### Claim 10 (depends on 1) The method of claim 1, wherein each cascading model policy utilizes a respective relevance associated with each sensor, and wherein each cascading model policy specifies accessing a first sensor signal from a first sensor associated with a first relevance before accessing a second sensor signal from a second sensor associated with a second relevance if the first relevance is higher than the second relevance. ### Claim 11 (depends on 10) The method of claim 10, wherein the respective relevance associated with each sensor is determined based on the activity. ### Claim 12 (depends on 1) The method of claim 1, wherein detecting the change in the context of the first user associated with the activity of the first user comprises: calculating, based on the one or more machine-learning models and the one or more sensor signals, a first confidence score associated with the change in the context of the first user associated with the activity of the first user; determining whether the first confidence score is greater than a threshold score; and based on the determining of whether the first confidence score is greater than the threshold score: if the first confidence score is greater than the threshold score, confirming the change in the context of the first user associated with the activity of the first user; else: accessing, from the client system associated with the first user, a plurality of sensor signals captured by the plurality of sensors of the client system, wherein the plurality of sensor signals are accessed based on the one or more cascading model policies; and calculating, based on the one or more machine-learning models and the plurality of sensor signals, a second confidence score associated with the change in the context of the first user associated with the activity of the first user, wherein if the second confidence score is greater than the threshold score, confirming the change in the context of the first user associated with the activity of the first user. ### Claim 13 (depends on 1) The method of claim 1, wherein the one or more cascading model policies specify, for each of a plurality of activities, a particular sequence of accessing sensor signals from the plurality of sensors, wherein additional sensors from the plurality of sensors are accessed based on confidence scores associated with detections of changes in the context. ### Claim 14 One or more computer-readable non-transitory non-volatile storage media embodying software that is operable when executed to: access, from a client system associated with a first user, one or more sensor signals captured by one or more sensors of the client system, wherein the client system comprises a plurality of sensors, and wherein the one or more sensors signals are accessed from the one or more sensors based on one or more cascading model policies, wherein each cascading model policy utilizes one or more of a respective cost or relevance associated with each sensor; detect, based on one or more machine-learning models and the one or more sensor signals, a change in a context of the first user associated with an activity of the first user, wherein the change in the context of the first user satisfies a trigger condition associated with the activity; and automatically capture, responsive to the detected change in the context of the first user, visual data by one or more cameras of the client system, wherein each cascading model policy utilizes a respective cost associated with each sensor, and wherein each cascading model policy specifies accessing a first sensor signal from a first sensor associated with a first cost before accessing a second sensor signal from a second sensor associated with a second cost if the first cost is lower than the second cost. ### Claim 15 (depends on 14) The media of claim 14, wherein the one or more sensor signals comprise one or more of an inertial measurement unit (IMU) signal, an audio signal, a GPS signal, an electromyography (EMG) signal, or a visual signal. ### Claim 16 A system comprising: one or more processors; and a non-transitory non-volatile memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to: access, from a client system associated with a first user, one or more sensor signals captured by one or more sensors of the client system, wherein the client system comprises a plurality of sensors, and wherein the one or more sensors signals are accessed from the one or more sensors based on one or more cascading model policies, wherein each cascading model policy utilizes one or more of a respective cost or relevance associated with each sensor; detect, based on one or more machine-learning models and the one or more sensor signals, a change in a context of the first user associated with an activity of the first user, wherein the change in the context of the first user satisfies a trigger condition associated with the activity; and automatically capture, responsive to the detected change in the context of the first user, visual data by one or more cameras of the client system, wherein each cascading model policy utilizes a respective cost associated with each sensor, and wherein each cascading model policy specifies accessing a first sensor signal from a first sensor associated with a first cost before accessing a second sensor signal from a second sensor associated with a second cost if the first cost is lower than the second cost. ### Claim 17 (depends on 16) The system of claim 16, wherein the one or more sensor signals comprise one or more of an inertial measurement unit (IMU) signal, an audio signal, a GPS signal, an electromyography (EMG) signal, or a visual signal.