# US 10,372,735 B2 · Generating activity summaries > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,735 B2 - Document ID: US10372735B2 - Status: Granted - Kind: B2 - Application number: 14/718,005 - Filed: 2015-05-20 - Published: 2015-11-26 - Granted: 2019-08-06 - Earliest claimed: 2014-05-20 - Assignees: Google Inc. - Applicants: Google Inc. - Inventors: Joseph Robert Smarr; Anand Agarawala; Brett Rolston Lider; Benjamin David Eidelson - Web page: https://patentgrep.com/patent/US10372735B2 ## CPC - G06Q 10/40: Business processes related to social networking or social networking services - G06F 16/26: Visual data mining; Browsing structured data - G06F 16/287: {Visualization; Browsing} - G06F 16/44: Browsing; Visualisation therefor - G06F 16/955: using information identifiers, e.g. uniform resource locators [URL] - H04L 65/403: Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15) - H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A system and method for generating activity summaries for users. User activity information is received by a user activity information receiver module. Once enough data is received and processed, the data is analyzes and segmented to determine and create an activity summary or story. Content is selected. The selection of the content includes selection of content items, such as multimedia items, e.g. pictures and videos. Secondary information, such as user activity information or location information is analyzed. A story is generated based on the selected content, including the selected content. The story is represented by a display of the selected media and other information associated with the media. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, using one or more computing devices, first activity information associated with a first activity of a first user targeting a first content item; receiving, using the one or more computing devices, second activity information associated with a second activity of the first user targeting a second content item; autonomously determining, from an analysis of the first content item within a plurality of content items, a first change in location of the first user by comparing a location associated with the first activity information with a reference location, wherein the determined first change in location indicates a beginning of a story to be visually represented by an activity summary including a plurality of activities of the first user; autonomously determining, from an analysis of the second content item within the plurality of content items, a second change in location of the first user, based on a location associated with the second activity information, wherein the determined second change in location indicates an ending of the story; autonomously determining timeline content items of the plurality of content items that are associated with activities of the first user between a time of the first activity information associated with the beginning of the story and a time of the second activity information associated with the ending of the story; ranking the timeline content items according to a relevance of the timeline content items to the first user or relevance of the timeline content items to one or more contacts of the first user; autonomously selecting a set of content items of the timeline content items for inclusion in the activity summary that represents the story, based on at least one of: the ranking of the timeline content items, time information associated with the timeline content items, location information associated with the timeline content items, or a quality of the timeline content items; generating the activity summary based on the set of content items selected for inclusion in the activity summary, wherein the activity summary is a summary of one or more activities of the first user; generating a visual representation of the activity summary; and sending, using the one or more computing devices, the visual representation of the activity summary for display. ### Claim 2 (depends on 1) The computer-implemented method of claim 1 wherein the plurality of content items includes at least one multimedia item. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: autonomously enriching the activity summary by obtaining supplemental information based on recognition of content or a location associated with the activity summary and adding the supplemental information to the activity summary, wherein the supplemental information includes one or more of additional content and additional information. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: autonomously determining a title associated with the activity summary. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: ranking activity information associated with the activities of the first user. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the first activity information includes at least one of: a unique user identifier associated with the first user, a predetermined verb describing the first activity of the first user, or a target identifier identifying a first content item as a target of the first activity of the first user, wherein the second activity information includes at least one of: the unique user identifier associated with the first user, a predetermined verb describing the second activity of the first user, or a target identifier identifying a second content item as a target of the second activity of the first user; and further comprising: autonomously categorizing activity information associated with activities of the first user, including the first activity information associated with the first activity of the first user and the second activity information associated with the second activity of the first user, wherein the categorizing is based on at least one of: the unique user identifier, predetermined verbs describing the activities including the predetermined verb describing the first activity and the predetermined verb describing the second activity, target identifiers identifying content items as targets of activities including the target identifier identifying a first content item and the target identifier identifying a second content item, or a commonality; and autonomously grouping a subset of categorized activity information associated with the activities of the first user into a first group of categorized activity information based on the first change in location of the first user indicating the beginning of the story, the second change in location of the first user indicating the ending of the story, and the categorization, wherein the activity summary further includes the first group of categorized activity information associated with the activities of the first user. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving input from the first user modifying the activity summary. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: autonomously generating a language-agnostic title using a template; and assigning the language-agnostic title to the story. ### Claim 9 A computer-implemented method comprising: receiving, using one or more computing devices, first activity information associated with a first activity of a first user, the first activity information including a unique user identifier associated with the first user, a predetermined verb describing the first activity of the first user, and a target identifier identifying a first content item as a target of the first activity of the first user; receiving, using the one or more computing devices, second activity information associated with a second activity of the first user, the second activity information including the unique user identifier associated with the first user, a predetermined verb describing the second activity of the first user, and a target identifier identifying a second content item as a target of the second activity of the first user; categorizing activity information associated with activities of the first user, including the first activity information associated with the first activity of the first user and the second activity information associated with the second activity of the first user, wherein the categorizing is based on the unique user identifier, predetermined verbs describing each activity including the predetermined verb describing the first activity and the predetermined verb describing the second activity, target identifiers identifying content items including the target identifier identifying the first content item, and the target identifier identifying the second content item, and a commonality identified among at least two of the activity information; autonomously determining, from an analysis of the first content item within a plurality of content items, a first change in a location of the first user by comparing a location associated with the first activity information with a reference location, wherein the determined first change in location indicates a beginning of a story to be visually represented by an activity summary including a plurality of activities of the first user; autonomously determining, from an analysis of the second content item within the plurality of content items, a second change in location of the first user, based on a location associated with the second activity information, wherein the determined second change in location indicates an ending of the story; grouping a subset of the categorized activity information associated with the activities of the first user into a first group of categorized activity information based on the first change in location of the first user indicating the beginning of the story, the second change in location of the first user indicating the ending of the story, and the categorization; autonomously selecting, a plurality of timeline content items for inclusion in the activity summary that are associated with activities of the first user between a time of the first activity information associated with the beginning of the story and a time of the second activity information associated with the ending of the story; generating the activity summary that includes the plurality of timeline content items and the first group of the categorized activity information, wherein the activity summary is a summary of one or more activities of the first user; generating a visual representation of the activity summary; and sending, using the one or more computing devices, the visual representation of the activity summary for display. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, further comprising: ranking the first group of categorized activity information based on relevance to the first user and selecting of the plurality of select content items is further based on the ranking. ### Claim 11 A computer program product comprising a non-transitory computer usable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to: receive first activity information associated with a first activity of a first user targeting a first content item; receive second activity information associated with a second activity of the first user targeting a second content item; autonomously determine, from an analysis of the first content item within a plurality of content items, a first change in a location of the first user by comparing a location associated with the first activity information with a reference location, wherein the determined first change in location indicates a beginning of a story to be visually represented by an activity summary including a plurality of activities of the first user; autonomously determine, from an analysis of the second content item within the plurality of content items, a second change in location of the first user, based on a location associated with the second activity information, wherein the determined second change in location indicates an ending of the story; autonomously determine timeline content items of the plurality of content items that are associated with activities of the first user between a time of the first activity information associated with the beginning of the story and the second activity information associated with the ending of the story; ranking the timeline content items according to a relevance of the timeline content items to the first user or relevance of the timeline content items to one or more contacts of the first user; autonomously selecting a set of content items of the timeline content items for inclusion in the activity summary that represents the story, based on at least one of: the ranking of the timeline content items, time information associated with the timeline content items, location information associated with the timeline content items, or a quality of the timeline content items; generate the activity summary that includes the set of content items, wherein the activity summary is a summary of one or more activities of the first user; generate a visual representation of the activity summary; and send the visual representation of the activity summary for display. ### Claim 12 A system comprising: a processor; and a memory storing instructions that, when executed, cause the system to: receive first activity information associated with a first activity of a first user targeting a first content item; receive second activity information associated with a second activity of the first user targeting a second content item; autonomously determine, from an analysis of the first content item within a plurality of content items, a first change in a-location of the first user by comparing a location associated with the first activity information with a reference location, wherein the determined first change in location indicates a beginning of a story to be visually represented by an activity summary including a plurality of activities of the first user; autonomously determine, from an analysis of the second content item within the plurality of content items, a second change in location of the first user, based on a location associated with the second activity information, wherein the determined second change in location indicates an ending of the story; autonomously determine timeline content items of the plurality of content items that are associated with the activities of the first user between a time of the first activity information associated with the beginning of the story and a time of the second activity information associated with the ending of the story; ranking the timeline content items according to a relevance of the timeline content items to the first user or relevance of the timeline content items to one or more contacts of the first user; autonomously selecting a set of content items of the timeline content items for inclusion in the activity summary that represents the story, based on at least one of: the ranking of the timeline content items, time information associated with the timeline content items, location information associated with the timeline content items, or a quality of the timeline content items; generate the activity summary, wherein the activity summary is a summary of one or more activities of the first user; generate a visual representation of the activity summary; and send the visual representation of the activity summary for display. ### Claim 13 (depends on 12) The system of claim 12 wherein the plurality of content items includes at least one multimedia item. ### Claim 14 (depends on 12) The system of claim 12 comprising instructions that, when executed, cause the system to: autonomously enrich the activity summary by obtaining supplemental information based on recognition of content or a location associated with the activity summary and adding the supplemental information to the activity summary, wherein the supplemental information includes one or more of additional content and additional information. ### Claim 15 (depends on 12) The system of claim 12, wherein the first activity information includes at least one of: a unique user identifier associated with the first user, a predetermined verb describing the first activity of the first user, or a target identifier identifying a first content item as a target of the first activity of the first user, wherein the second activity information includes at least one of: the unique user identifier associated with the first user, a predetermined verb describing the second activity of the first user, or a target identifier identifying a second content item as a target of the second activity of the first user; and further comprising instructions that, when executed, cause the system to: autonomously categorize the activity information associated with activities of the first user, including the first activity information associated with the first activity of the first user and second activity information associated with the second activity of the first user, wherein the categorizing is based on at least one of: the unique user identifier, predetermined verbs describing the activities including the predetermined verb describing the first activity and the predetermined verb describing the second activity, target identifiers identifying content items as targets of activities including the target identifier identifying a first content item and the target identifier identifying a second content item, or a commonality; and autonomously group a subset of categorized activity information associated with the activities of the first user into a first group of the categorized activity information based on the first change in location of the first user indicating the beginning of the story, the second change in location of the first user indicating the ending of the story, and the categorization, wherein the set of content items is selected from the first group of the categorized activity information associated with the activities of the first user. ### Claim 16 (depends on 12) The system of claim 12 comprising instructions that, when executed, cause the system to: autonomously generate a language-agnostic title using a template; and assign the language-agnostic title to the story.