# US 12,008,318 B2 · Automatic personalized story generation for visual media > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,318 B2 - Document ID: US12008318B2 - Status: Granted - Kind: B2 - Application number: 17/069,263 - Filed: 2020-10-13 - Published: 2021-01-28 - Granted: 2024-06-11 - Earliest claimed: 2017-12-21 - Assignees: Meta Platforms, Inc. - Applicants: Meta Platforms, Inc. - Inventors: Ying Zhang; Shengbo Guo - Web page: https://patentgrep.com/patent/US12008318B2 ## CPC - G06F 40/237: Lexical tools - G06F 40/10: Text processing (natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40) - G06F 40/56: Natural language generation - G06Q 10/48: using social graphs - G06V 20/30: in albums, collections or shared content, e.g. social network photos or video - G06V 20/47: {Detecting features for summarising video content} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/2743: Video hosting of uploaded data from client - H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)} - H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)} - H04N 21/854: Content authoring - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Exemplary embodiments relate to the automatic generation of captions for visual media, including photos, photo albums, non-live video, and live video. The visual media may be analyzed to determine contextual information (such as location information, people and objects in the video, time, etc.). A system may integrate this information with information from the user's social network and a personalized language model built using public-facing language from the user. The personalized language model captures the user's way of speaking to make the generated captions more detailed and personalized. The language model may account for the context in which the video was generated. The captions maybe used to simplify and encourage content generation, and may also be used to index visual media, rank the media, and recommend the media to users likely to engage with the media. ## Claims ### Claim 1 A method, comprising: receiving a request to generate a personalized language model configured to reflect a personal narrative style of a user; retrieving public-facing language of the user; using the public-facing language to build the personalized language model, the personalized language model reflecting a vocabulary, a sentence or phrase structure, and a sentence construction specific to the user; accessing visual media; analyzing information associated with the visual media to identify a context of the visual media; providing the context to the personalized language model; generating a caption for the visual media in the user's personal narrative style using the personalized language model and the context; and retrieving some public-facing language that is machine-generated language, and the machine-generated language is excluded from use when building the personalized language model. ### Claim 2 (depends on 1) The method of claim 1, wherein the request to generate the personalized language model includes representative target language, and the representative target language is used to supplement the retrieved public facing language. ### Claim 3 A method comprising: receiving a request to generate a personalized language model configured to reflect a personal narrative style of a user; retrieving public-facing language of the user; using the public-facing language to build the personalized language model, the personalized language model reflecting a vocabulary, a sentence or phrase structure, and a sentence construction specific to the user; accessing visual media; analyzing information associated with the visual media to identify a context of the visual media; providing the context to the personalized language model; generating a caption for the visual media in the user's personal narrative style using the personalized language model and the context; retrieving some public-facing language associated with the user that is machine-generated language; determining through extrinsic evidence that the user approves of the machine-generated language; and including the machine-generated language when building the personalized language model. ### Claim 4 (depends on 1) The method of claim 1, further comprising: determining that the retrieved public-facing language is insufficient to build the personalized language model; identifying one or more associated users that are connected to the user in a social networking graph; and supplementing the retrieved public-facing language with public-facing language of the one or more associated users. ### Claim 5 (depends on 1) The method of claim 1, wherein the public-facing language includes captions and one or more other items, and building the personalized language model comprises weighting the captions more heavily than the one or more other items. ### Claim 6 (depends on 1) The method of claim 1, wherein building the personalized language model comprises: outputting a test caption from the personalized language model; assigning a confidence score to the test caption based on one or more of whether the caption makes grammatical sense, whether the personalized language model was able to find suitable user-specific vocabulary, or whether the personalized language model was able to identify a suitable user-specific sentence structure; comparing the confidence score to a predetermined threshold; and when the confidence score does not exceed the predetermined threshold, retrieving supplemental public-facing language to train the personalized language model. ### Claim 7 A non-transitory computer-readable medium storing instructions that, when executed, cause a processor to: receive a request to generate a personalized language model configured to reflect a personal narrative style of a user; retrieve public-facing language of the user; use the public-facing language to build the personalized language model, the personalized language model reflecting a vocabulary, a sentence or phrase structure, and a sentence construction specific to the user; access visual media; analyze information associated with the visual media to identify a context of the visual media; provide the context to the personalized language model; generate a caption for the visual media in the user's personal narrative style using the personalized language model and the context; and store instructions for retrieving some public-facing language associated with the user that is machine-generated language, and the machine-generated language is excluded from use when building the personalized language model. ### Claim 8 (depends on 7) The medium of claim 7, wherein the request to generate the personalized language model includes representative target language, and the representative target language is used to supplement the retrieved public facing language. ### Claim 9 A non-transitory computer-readable medium storing instructions that, when executed, cause a processor to: receive a request to generate a personalized language model configured to reflect a personal narrative style of a user; retrieve public-facing language of the user; use the public-facing language to build the personalized language model, the personalized language model reflecting a vocabulary, a sentence or phrase structure, and a sentence construction specific to the user; access visual media; analyze information associated with the visual media to identify a context of the visual media; provide the context to the personalized language model; generate a caption for the visual media in the user's personal narrative style using the personalized language model and the context; retrieve some public-facing language associated with the user that is machine-generated language; determine through extrinsic evidence that the user approves of the machine-generated language; and include the machine-generated language when building the personalized language model. ### Claim 10 (depends on 7) The medium of claim 7, further storing instructions for: determining that the retrieved public-facing language is insufficient to build the personalized language model; identifying one or more associated users that are connected to the user in a social networking graph; and supplementing the retrieved public-facing language with public-facing language of the one or more associated users. ### Claim 11 (depends on 7) The medium of claim 7, wherein the public-facing language includes captions and one or more other items, and building the personalized language model comprises weighting the captions more heavily than the one or more other items. ### Claim 12 (depends on 7) The medium of claim 7, wherein building the personalized language model comprises: outputting a test caption from the personalized language model; assigning a confidence score to the test caption based on one or more of whether the caption makes grammatical sense, whether the personalized language model was able to find suitable user-specific vocabulary, or whether the personalized language model was able to identify a suitable user-specific sentence structure; comparing the confidence score to a predetermined threshold; and when the confidence score does not exceed the predetermined threshold, retrieving supplemental public-facing language to train the personalized language model. ### Claim 13 An apparatus comprising: a hardware processor; and a non-transitory computer-readable medium storing instructions that, when executed, cause the processor to: receive a request to generate a personalized language model configured to reflect a personal narrative style of a user; retrieve public-facing language of the user; use the public-facing language to build the personalized language model, the personalized language model reflecting a vocabulary, a sentence or phrase structure, and a sentence construction specific to the user; access visual media; analyze information associated with the visual media to identify a context of the visual media; provide the context to the personalized language model; generate a caption for the visual media in the user's personal narrative style using the personalized language model and the context; and retrieve some public-facing language associated with the user that is machine-generated language, wherein the machine-generated language is excluded from use when building the personalized language model. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the request to generate the personalized language model includes representative target language, and the representative target language is used to supplement the retrieved public facing language. ### Claim 15 (depends on 13) The apparatus of claim 13, wherein the medium further stores instructions for: determining through extrinsic evidence that the user approves of the machine-generated language; and including the machine-generated language when building the personalized language model. ### Claim 16 An apparatus comprising: a hardware processor; and a non-transitory computer-readable medium storing instructions that, when executed, cause the processor to: receive a request to generate a personalized language model configured to reflect a personal narrative style of a user; retrieve public-facing language of the user; use the public-facing language to build the personalized language model, the personalized language model reflecting a vocabulary, a sentence or phrase structure, and a sentence construction specific to the user; access visual media; analyze information associated with the visual media to identify a context of the visual media; provide the context to the personalized language model; generate a caption for the visual media in the user's personal narrative style using the personalized language model and the context; retrieve some public-facing language associated with the user that is machine-generated language; determine that the retrieved public-facing language is insufficient to build the personalized language model; identify one or more associated users that are connected to the user in a social networking graph; and supplement the retrieved public-facing language with public-facing language of the one or more associated users. ### Claim 17 (depends on 13) The apparatus of claim 13, wherein the public-facing language includes captions and one or more other items, and building the personalized language model comprises weighting the captions more heavily than the one or more other items.