# US 12,008,485 B2 · Identifying physical activities performed by a user of a computing device based on media consumption > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,485 B2 - Document ID: US12008485B2 - Status: Granted - Kind: B2 - Application number: 18/192,783 - Filed: 2023-03-30 - Published: 2023-07-27 - Granted: 2024-06-11 - Earliest claimed: 2019-12-23 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Ant Oztaskent; Flavio Lerda - Web page: https://patentgrep.com/patent/US12008485B2 ## CPC - G06N 5/04: Inference or reasoning models - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser - G06N 20/00: Machine learning - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)} - H04N 21/44222: {Analytics of user selections, e.g. selection of programmes or purchase activity (monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33)} - H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)} - G06V 2201/10: Recognition assisted with metadata - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method includes identifying, based on sensor data received by a motion sensor, a physical activity performed by a user of the computing system during a time period and determining whether the user consumed media during the time period that the user performed the physical activity. The method also includes responsive to determining that the user consumed the media during the time period that the user performed the physical activity, determining, based on data indicative of the media consumed by the user, an updated physical activity performed by the user during the time period; and outputting data indicating the updated physical activity. ## Claims ### Claim 1 A method comprising: determining, by a computing system, an initial prediction of a physical activity performed by a user of the computing system; determining, by the computing system, whether the user consumed media during performance of the physical activity; responsive to determining that the user consumed the media during performance the physical activity, determining, by the computing system, based on data indicative of the media consumed by the user, an updated prediction of the physical activity; and outputting, by the computing system, data indicating the updated prediction. ### Claim 2 (depends on 1) The method of claim 1, wherein the media consumed by the user includes a video viewed by the user, and wherein determining the updated prediction comprises: determining, by the computing system, a physical activity associated with the video viewed by the user during performance of the physical activity; and determining, by the computing system, that the updated prediction of the physical activity performed includes the physical activity associated with the video that was viewed by the user performance of the physical activity. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the updated prediction comprises: determining, based on the data indicative of the media, for a plurality of portions of the media, a predicted sub-activity performed by the user during respective portions of the plurality of portions of the media. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the initial prediction of the physical activity performed by the user comprises determining, by the computing system, based on a machine-trained model that classifies physical activities using sensor data received from a motion sensor, the initial prediction of the physical activity performed by the user, and wherein the method further comprises, responsive to determining the updated prediction, updating, by the computing system, based on the sensor data received and the updated prediction, the machine-trained model. ### Claim 5 (depends on 1) The method of claim 1, wherein the computing system includes a user profile indicating, for each media of a plurality of media consumed by the user, a time period during which the user consumed the respective media, and wherein determining whether the user consumed the media comprises: querying, by the computing system, the user profile to determine whether the user profile includes data indicating media consumed by the user during performance of the physical activity. ### Claim 6 (depends on 1) The method of claim 1, wherein determining whether the user consumed the media comprises: determining, by the computing system, based on audio input data captured by a microphone and a set of fingerprints that are each associated with a respective media, whether the user consumed the media. ### Claim 7 (depends on 1) The method of claim 1, further comprising: identifying, by the computing system, one or more physical activities associated with the media by applying a machine trained model to at least one of the media or metadata for the media. ### Claim 8 (depends on 1) The method of claim 1, further comprising: responsive to determining that the user consumed the media during performance of the physical activity, updating, by the computing system, based on the media consumed by the user, a machine-trained model that identifies fitness media. ### Claim 9 (depends on 8) The method of claim 8, wherein the media consumed by the user includes a video viewed by the user, and wherein the machine-trained model that identifies fitness media further identifies an activity associated with the video. ### Claim 10 A computing device comprising: at least one processor; and memory comprising instructions that, when executed by the at least one processor, cause the at least one processor to: determine an initial prediction of a physical activity performed by a user of the computing device; determine whether the user consumed media during performance of the physical activity; responsive to determining that the user consumed the media during performance the physical activity, determine, based on data indicative of the media consumed by the user, an updated prediction of the physical activity; and output data indicating the updated prediction. ### Claim 11 (depends on 10) The computing device of claim 10, wherein the media consumed by the user includes a video viewed by the user, and wherein the instruction that cause the at least one processor to determine the updated prediction comprise instructions that cause the processor to: determine a physical activity associated with the video viewed by the user during performance of the physical activity; and determine that the updated prediction of the physical activity performed by the user includes the physical activity associated with the video that was viewed by the user. ### Claim 12 (depends on 10) The computing device of claim 10, wherein the media consumed by the user includes a video viewed by the user, and wherein the instructions that cause the at least one processor to determine the updated physical activity comprise instructions that cause the processor to: determine, based on the data indicative of the media, for a plurality of portions of the media, a predicted sub-activity performed by the user during respective portions of the plurality of portions of the media. ### Claim 13 (depends on 10) The computing device of claim 10, wherein the instructions further cause the at least one processor to determine the initial prediction of the physical activity performed by the user by at least causing the at least one processor to determine, based on a machine-trained model that classifies physical activities using sensor data received from a motion sensor, the initial prediction of the physical activity performed by the user, and wherein execution of the instructions further causes the at least one processor to, responsive to determining the updated prediction of the physical activity, update, based on the sensor data and the updated prediction of the physical activity, the machine-trained model. ### Claim 14 (depends on 10) The computing device of claim 10, wherein the computing device includes a user profile indicating, for each media of a plurality of media consumed by the user, a time period during which the user consumed the respective media, and wherein the instructions that cause the at least one processor to determine whether the user consumed the media comprise instructions that cause the at least one processor to: query the user profile to determine whether the user profile includes data indicating media consumed by the user during performance of the physical activity. ### Claim 15 (depends on 10) The computing device of claim 10, wherein execution of the instructions further causes the at least one processor to: responsive to determining that the user consumed the media during performance of the physical activity, update, based on the media consumed by the user during performance of the physical activity, a machine-trained model that identifies fitness media. ### Claim 16 A computer-readable storage medium comprising instructions that, when executed by at least one processor of a computing device, cause the at least one processor to: determine an initial prediction of a physical activity performed by a user of the computing device; determine whether the user consumed media during performance of the physical activity; responsive to determining that the user consumed the media during performance the physical activity, determine, based on data indicative of the media consumed by the user, an updated prediction of the physical activity; and output data indicating the updated prediction. ### Claim 17 (depends on 16) The computer-readable storage medium of claim 16, wherein determining the initial prediction of the physical activity performed by the user comprises determining, by the computing device, based on sensor data received from a motion sensor, the initial prediction of the physical activity performed by the user. ### Claim 18 (depends on 16) The computer-readable storage medium of claim 16, wherein determining the updated prediction of the physical activity comprises: determining a physical activity associated with the media consumed during performance the physical activity that is different than the initial prediction of the physical activity; and determining the updated prediction of the physical activity as the physical activity associated with the media.