# US 2019/0318003 A1 · System and method for associating textual summaries with content media > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318003 A1 - Document ID: US20190318003A1 - Status: Published application - Kind: A1 - Application number: 15/953,314 - Filed: 2018-04-13 - Published: 2019-10-17 - Earliest claimed: 2018-04-13 - Assignees: Fuji Xerox Co., Ltd. - Applicants: Fuji Xerox Co., Ltd. - Inventors: Lyndon KENNEDY; Francine CHEN; Yin-Ying CHEN; David Ayman SHAMMA - Web page: https://patentgrep.com/patent/US20190318003A1 ## CPC - G06F 16/41: Indexing; Data structures therefor; Storage structures - G06F 16/48: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06F 16/285: {Clustering or classification} - G06F 16/345: {Summarisation for human users} - G06F 16/438: Presentation of query results - G06F 16/55: Clustering; Classification - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method and system of associating textual summaries with data representative of media content is provided. The method may include receiving a plurality of textual summaries, each textual summary representative of an event, pairing, by a neural network, each received textual summary with each of a plurality of pieces of data, each piece of data representative of media content, to generate a plurality of text-data pairings; and associating a first selected textual summary with a first piece of data based on a similarity of content features extracted from each received textual summary to content features extracted from each piece of data in each of the plurality of text-data pairings. ## Claims ### Claim 1 A method of associating textual summaries with pieces of data representative of media content, the method comprising: receiving a plurality of textual summaries, each textual summary representative of an event; pairing, by a neural network, each received textual summary with each of a plurality of pieces of data, each piece of data representative of media content, to generate a plurality of text-data pairings; and associating a first textual summary selected from the plurality of textual summaries with a first piece of data selected from the plurality of pieces of data based on a similarity of content features extracted from each received textual summary to content features extracted from each piece of data in each of the plurality of text-data pairings. ### Claim 2 (depends on 1) The method of claim 1, further comprising associating a second textual summary selected from the plurality of textual summaries with a second piece of data selected the plurality of pieces of data based on the similarity of content features extracted from each received text summary to content features extracted from each piece of data in each of the plurality of text-data pairings, wherein the first piece of data is excluded from being associated with the second textual summary; and generating a media content album based on the association between the first textual summary and the first piece of data and the association between the second textual summary and the second piece of data. ### Claim 3 (depends on 2) The method of claim 2, wherein the associating the first textual summary with the first piece of data is based on the similarity of content features extracted from the first piece of data to the first textual summary being the highest similarity of any of the plurality of text-data pairings; and wherein the associating the second textual summary with the second piece of data is based on the similarity of content features extracted from the second piece of data to the second textual summary being the second highest similarity of any of the plurality of text-data pairings. ### Claim 4 (depends on 2) The method of claim 2, wherein the associating the first textual summary with the first piece of data is based on the similarity of content features extracted from the first piece of data to the first textual summary being the highest similarity of a first subset of text-data pairs selected from the plurality of text-data pairings, the first subset comprising only text-data pairs in which the first textual summary has been paired to each of the plurality of pieces of data; and wherein the associating the second textual summary with the second piece of data is based on the similarity of content features extracted from the second piece of data to the second textual summary being the highest similarity of a second subset of text-data pairs selected from the plurality of text-data pairings, the second subset comprising only text-data pairs in which the second textual summary has been paired to each of the plurality of pieces of data. ### Claim 5 (depends on 1) The method of claim 1, wherein the associating the first textual summary with the first piece of data comprises: clustering a first subset of the plurality of pieces of data based on content features extracted from each of the pieces of data in the subset to form a first data cluster; determining a similarity between each of the pieces of data in the first data cluster and the first textual summary; determining a first aggregate similarity score for the first data cluster based on the determined similarity between each of the pieces of data in the first data cluster and the first textual summary; and associating the first data cluster with the first textual summary based on the first aggregate similarity score; and wherein the associating the second textual summary with the second piece of data comprises: clustering a second subset of the plurality of pieces of data based on content features extracted from each of the pieces of data in the subset to form a second data cluster; determining a similarity between each of the pieces of data in the second data cluster and the second textual summary; determining a second aggregate similarity score for the second data cluster based on the determined similarity between each of the pieces of data in the second data cluster and the second textual summary; and associating the second data cluster with the second textual summary based on the second aggregate similarity score. ### Claim 6 (depends on 2) The method of claim 2, wherein the similarity of content features extracted from each received textual summary to content features extracted from each piece of data in each of the plurality of text-data pairings is calculated based on a user profile associated with a user requesting the association of the textual summaries with the pieces of data representative of media content. ### Claim 7 (depends on 6) The method of claim 6, wherein the each of the plurality of the textual summaries is generated based on a social media post associated with a user account associated with the user requesting the association of the textual summaries with the pieces of data representative of media content. ### Claim 8 (depends on 2) The method of claim 2, wherein the similarity of content features extracted from each received textual summary to content features extracted from each piece of data in each of the plurality of text-data pairings is calculated based on at least one capture setting associated with a media capture device used to capture at least one of the plurality of the pieces of data representative of media content. ### Claim 9 A non-transitory computer readable medium having stored therein a program for making a computer execute a method of associating textual summaries with pieces of data representative of media content, the method comprising: receiving a plurality of textual summaries, each textual summary representative of an event; pairing, by a neural network, each received textual summary with each of a plurality of pieces of data, each piece of data representative of media content, to generate a plurality of text-data pairings; associating a first textual summary selected from the plurality of textual summaries with a first piece of data selected from the plurality of pieces of data based on a similarity of content features extracted from each received textual summary to content features extracted from each piece of data in each of the plurality of text-data pairings. ### Claim 10 (depends on 9) The non-transitory computer readable medium of claim 9, further comprising associating a second textual summary selected from the plurality of textual summaries with a second piece of data selected the plurality of pieces of data based on the similarity of content features extracted from each received text summary to content features extracted from each piece of data in each of the plurality of text-data pairings, wherein the first piece of data is excluded from being associated with the second textual summary; and generating a media content album based on the association between the first textual summary and the first piece of data and the association between the second textual summary and the second piece of data. ### Claim 11 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the associating the first textual summary with the first piece of data is based on the similarity of content features extracted from the first piece of data to the first textual summary being the highest similarity of any of the plurality of text-data pairings; and wherein the associating the second textual summary with the second piece of data is based on the similarity of content features extracted from the second piece of data to the second textual summary being the second highest similarity of any of the plurality of text-data pairings. ### Claim 12 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the associating the first textual summary with the first piece of data is based on the similarity of content features extracted from the first piece of data to the first textual summary being the highest similarity of a first subset of text-data pairs selected from the plurality of text-data pairings, the first subset comprising only text-data pairs in which the first textual summary has been paired to each of the plurality of pieces of data; and wherein the associating the second textual summary with the second piece of data is based on the similarity of content features extracted from the second piece of data to the second textual summary being the highest similarity of a second subset of text-data pairs selected from the plurality of text-data pairings, the second subset comprising only text-data pairs in which the second textual summary has been paired to each of the plurality of pieces of data. ### Claim 13 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the associating the first textual summary with the first piece of data comprises: clustering a first subset of the plurality of pieces of data based on content features extracted from each of the pieces of data in the subset to form a first data cluster; determining a similarity between each of the pieces of data in the first data cluster and the first textual summary; determining a first aggregate similarity score for the first data cluster based on the determined similarity between each of the pieces of data in the first data cluster and the first textual summary; and associating the first data cluster with the first textual summary based on the first aggregate similarity score; and wherein the associating the second textual summary with the second piece of data comprises: clustering a second subset of the plurality of pieces of data based on content features extracted from each of the pieces of data in the subset to form a second data cluster; determining a similarity between each of the pieces of data in the second data cluster and the second textual summary; determining a second aggregate similarity score for the second data cluster based on the determined similarity between each of the pieces of data in the second data cluster and the second textual summary; and associating the second data cluster with the second textual summary based on the second aggregate similarity score. ### Claim 14 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the similarity of content features extracted from each received textual summary to content features extracted from each piece of data in each of the plurality of text-data pairings is calculated based on a user profile associated with a user requesting the association of the textual summaries with the pieces of data representative of media content. ### Claim 15 (depends on 14) The non-transitory computer readable medium of claim 14, wherein the each of the plurality of the textual summaries is generated based on a social media post associated with a user account associated with the user requesting the association of the textual summaries with the pieces of data representative of media content. ### Claim 16 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the similarity of content features extracted from each received textual summary to content features extracted from each piece of data in each of the plurality of text-data pairings is calculated based on at least one capture setting associated with a media capture device used to capture at least one of the plurality of the pieces of data representative of media content. ### Claim 17 A computer apparatus configured to associate textual summaries with pieces of data representative of media content, the computer apparatus comprising: a memory storing a plurality of pieces of data representative of media content; and a processor executing a process of visualizing recommended pieces of media content based on a viewed piece of media content, the process comprising: providing a user with an interface to submit a plurality of textual summaries, each textual summary representative of an event; pairing, by a neural network, each provided textual summary with each of the plurality of pieces of data stored in memory to generate a plurality of text-data pairings; associating a first textual summary selected from the plurality of textual summaries with a first piece of data selected from the plurality of pieces of data based on a similarity of content features extracted from each received textual summary to content features extracted from each piece of data in each of the plurality of text-data pairings. ### Claim 18 (depends on 17) The computer apparatus of claim 17, wherein the processor is further configured to associate a second textual summary selected from the plurality of textual summaries with a second piece of data selected the plurality of pieces of data based on the similarity of content features extracted from each received text summary to content features extracted from each piece of data in each of the plurality of text-data pairings, wherein the first piece of data is excluded from being associated with the second textual summary; and provide a media content album to the user based on the association between the first textual summary and the first piece of data and the association between the second textual summary and the second piece of data. ### Claim 19 (depends on 16) The computer apparatus of claim 16, wherein the associating the first textual summary with the first piece of data is based on the similarity of content features extracted from the first piece of data to the first textual summary being the highest similarity of any of the plurality of text-data pairings; and wherein the associating the second textual summary with the second piece of data is based on the similarity of content features extracted from the second piece of data to the second textual summary being the second highest similarity of any of the plurality of text-data pairings. ### Claim 20 (depends on 16) The computer apparatus of claim 16, wherein the associating the first textual summary with the first piece of data comprises: clustering a first subset of the plurality of pieces of data based on content features extracted from each of the pieces of data in the subset to form a first data cluster; determining a similarity between each of the pieces of data in the first data cluster and the first textual summary; determining a first aggregate similarity score for the first data cluster based on the determined similarity between each of the pieces of data in the first data cluster and the first textual summary; and associating the first data cluster with the first textual summary based on the first aggregate similarity score; and wherein the associating the second textual summary with the second piece of data comprises: clustering a second subset of the plurality of pieces of data based on content features extracted from each of the pieces of data in the subset to form a second data cluster; determining a similarity between each of the pieces of data in the second data cluster and the second textual summary; determining a second aggregate similarity score for the second data cluster based on the determined similarity between each of the pieces of data in the second data cluster and the second textual summary; and associating the second data cluster with the second textual summary based on the second aggregate similarity score.