# US 11,308,284 B2 · Smart cameras enabled by assistant systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,284 B2 - Document ID: US11308284B2 - Status: Granted - Kind: B2 - Application number: 16/659,363 - Filed: 2019-10-21 - Published: 2021-04-22 - Granted: 2022-04-19 - Earliest claimed: 2019-10-18 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Lisa Xiaoyi Huang; Eric Xiao; Nicholas Michael Andrew Benson; Yating Sheng; Zijian He - Web page: https://patentgrep.com/patent/US11308284B2 ## CPC - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 9/453: {Help systems} - G06F 9/485: {Task life-cycle, e.g. stopping, restarting, resuming execution (G06F9/4881 takes precedence)} - G06F 9/4862: {the task being a mobile agent, i.e. specifically designed to migrate} - G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues} - G06F 9/54: Interprogram communication - G06F 9/547: {Remote procedure calls [RPC]; Web services} - G06F 16/3329: Natural language query formulation - G06F 16/90332: {Natural language query formulation or dialogue systems} - G06F 16/9536: Search customisation based on social or collaborative filtering - G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions - G06F 40/126: Character encoding - G06F 40/205: Parsing - G06F 40/216: using statistical methods - G06F 40/242: Dictionaries - G06F 40/253: Grammatical analysis; Style critique - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/295: Named entity recognition - G06F 40/30: Semantic analysis - G06F 40/35: Discourse or dialogue representation - G06F 40/56: Natural language generation - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/08: Learning methods - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/09: Supervised learning - G06N 3/098: Distributed learning, e.g. federated learning - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 20/00: Machine learning - G06Q 10/00: Administration; Management - G06Q 10/109: Time management, e.g. calendars, reminders, meetings or time accounting - G06Q 10/1093: Calendar-based scheduling for persons or groups - G06Q 10/40: Business processes related to social networking or social networking services - G06Q 10/42: Determination of affinities or common interests between users - G06Q 10/48: using social graphs - G06Q 30/0603: {Catalogue creation or management} - G06Q 30/0631: {Recommending goods or services} - G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)} - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60) - G06V 20/20: in augmented reality scenes - G06V 20/30: in albums, collections or shared content, e.g. social network photos or video - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06V 40/25: {Recognition of walking or running movements, e.g. gait recognition} - G10L 15/063: {Training} - G10L 15/08: Speech classification or search - G10L 15/16: using artificial neural networks - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems - H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages - H04L 51/18: Commands or executable codes - H04L 51/212: using filtering or selective blocking - H04L 51/214: using selective forwarding - H04L 51/222: using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area - H04L 51/224: providing notification on incoming messages, e.g. pushed notifications of received messages - H04L 51/52: for supporting social networking services - H04L 67/306: User profiles - H04L 67/75: Indicating network or usage conditions on the user display - H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 2209/541: Client-server - G06N 3/048: Activation functions - G06N 3/084: Backpropagation, e.g. using gradient descent - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 40/174: {Facial expression recognition} - G06V 2201/10: Recognition assisted with metadata - G10L 13/00: Speech synthesis; Text to speech systems - G10L 15/07: to the speaker - G10L 2015/0631: {Creating reference templates; Clustering} - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/227: {of the speaker; Human-factor methodology} - G10L 2015/228: {of application context} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract In one embodiment, a method includes receiving a user input from a user from a client system associated with the user, wherein the client system comprises one or more cameras, determining one or more points of interest in a field of view of the one or more cameras based on one or more machine-learning models and sensory data captured by the one or more cameras, generating a plurality of media files based on the one or more points of interest, wherein each media file is a recording of at least one of the one or more points of interest, generating one or more highlight files based on the plurality of media files, wherein each highlight file comprises a media file that satisfies a predefined quality standard, and sending instructions for presenting the one or more highlight files to the client system. ## Claims ### Claim 1 A method comprising, by one or more computing systems: accessing sensory data captured by one or more cameras associated with a client system; determining, based on one or more machine-learning models and the sensory data captured by the one or more cameras, one or more points of interest in a field of view of the one or more cameras; generating, based on the one or more points of interest, a plurality of media files, wherein each media file is associated with a recording of at least one of the one or more points of interest; generating, based on the plurality of media files, one or more highlight files, wherein each highlight file comprises a media file that satisfies a predefined quality standard, and wherein each highlight file is associated with a respective captioning; and sending, to the client system, instructions for presenting the one or more highlight files. ### Claim 2 (depends on 1) The method of claim 1, wherein the sensory data is based on one or more of textual signals, visual signals, or audio signals. ### Claim 3 (depends on 1) The method of claim 1, further comprising: receiving, from the client system, a user input based on one or more of a text input, an audio input, an image input, a video input, an eye gaze, a gesture, or a motion, wherein determining the one or more points of interest in the field of view of the one or more cameras is responsive to the user input. ### Claim 4 (depends on 1) The method of claim 1, wherein each of the plurality of media files comprises one or more of an image or a video clip. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the points of interest comprises: detecting one or more people in the field of view; and determining, based on one or more facial recognition algorithms, one or more identifiers of one or more of the detected people. ### Claim 6 (depends on 5) The method of claim 5, wherein determining the points of interest is based on a measure of interestingness of one or more of the detected people, wherein the measure of interestingness is determined by the one or more machine-learning models based on one or more of the identifiers. ### Claim 7 (depends on 1) The method of claim 1, wherein determining the points of interest comprises: detecting one or more people in the field of view; and determining one or more facial expressions of one or more of the detected people. ### Claim 8 (depends on 7) The method of claim 7, wherein determining the points of interest is based on a measure of interestingness of one or more of the detected people, wherein the measure of interestingness is determined by the one or more machine-learning models based on one or more of the facial expressions. ### Claim 9 (depends on 1) The method of claim 1, wherein determining the points of interest comprises: detecting one or more objects in the field of view. ### Claim 10 (depends on 9) The method of claim 9, wherein determining the points of interest is based on a measure of interestingness of one or more of the detected objects, wherein the measure of interestingness is determined by the one or more machine-learning models based on one or more of the detected objects. ### Claim 11 (depends on 1) The method of claim 1, wherein determining the points of interest is based on eye gaze data of the user captured by the client system. ### Claim 12 (depends on 1) The method of claim 1, wherein the predefined quality standard is based on one or more of blurriness, lighting, or vividness of color. ### Claim 13 (depends on 1) The method of claim 1, further comprising: receiving, from the client system, a user query from the user in response to the highlight files; accessing a plurality of episodic memories associated with the user; identifying one or more episodic memories of the accessed episodic memories as related to the user query; retrieving one or more media files corresponding to the identified episodic memories, wherein each media file comprises one or more of a post, a comment, an image, or a video clip; and sending, to the client system, instructions for presenting the one or more media files corresponding to the identified episodic memories. ### Claim 14 (depends on 1) The method of claim 1, further comprising: sending, to the client system, instructions for zooming in one or more of the cameras to position one or more of the points of interest in a center of the field of view. ### Claim 15 (depends on 1) The method of claim 1, further comprising: sending, to the client system, instructions for zooming out one or more of the cameras to position one or more of the points of interest in a center of the field of view. ### Claim 16 (depends on 1) The method of claim 1, wherein the highlight files are personalized for the user based on one or more of: user profile data associated with the user; user preferences associated with the user; prior user inputs by the user; or user relationships with other users in a social graph. ### Claim 17 (depends on 1) The method of claim 1, further comprising: receiving, from the client system, a user request from the user to share one or more of the highlight files with one or more other users; and sending, to one or more other client systems associated with the one or more other users, respectively, instructions for presenting the shared highlight files. ### Claim 18 (depends on 1) The method of claim 1, further comprising: detecting a movement of the client system; and applying one or more visual stabilization algorithms to the sensory data captured by the one or more cameras. ### Claim 19 One or more computer-readable non-transitory storage media embodying software that is operable when executed to: access sensory data captured by one or more cameras associated with a client system; determine, based on one or more machine-learning models and the sensory data captured by the one or more cameras, one or more points of interest in a field of view of the one or more cameras; generate, based on the one or more points of interest, a plurality of media files, wherein each media file is associated with a recording of at least one of the one or more points of interest; generate, based on the plurality of media files, one or more highlight files, wherein each highlight file comprises a media file that satisfies a predefined quality standard, and wherein each highlight file is associated with a respective captioning; and send, to the client system, instructions for presenting the one or more highlight files. ### Claim 20 A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to: access sensory data captured by one or more cameras associated with a client system; determine, based on one or more machine-learning models and the sensory data captured by the one or more cameras, one or more points of interest in a field of view of the one or more cameras; generate, based on the one or more points of interest, a plurality of media files, wherein each media file is associated with a recording of at least one of the one or more points of interest; generate, based on the plurality of media files, one or more highlight files, wherein each highlight file comprises a media file that satisfies a predefined quality standard, and wherein each highlight file is associated with a respective captioning; and send, to the client system, instructions for presenting the one or more highlight files.