# US 2020/0401621 A1 · Cognitive video and audio search aggregation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401621 A1 - Document ID: US20200401621A1 - Status: Published application - Kind: A1 - Application number: 16/445,570 - Filed: 2019-06-19 - Published: 2020-12-24 - Earliest claimed: 2019-06-19 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Michael Bender; Michael P. Shute; Siddhartha Sood; Evelyn R. Anderson - Web page: https://patentgrep.com/patent/US20200401621A1 ## CPC - G06F 16/7837: {using objects detected or recognised in the video content} - G06F 16/24578: {using ranking} - G06F 16/71: Indexing; Data structures therefor; Storage structures - G06F 16/735: Filtering based on additional data, e.g. user or group profiles - G06F 16/738: Presentation of query results - G06F 16/784: {the detected or recognised objects being people} - G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation - G06F 18/40: Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor - G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space - G06V 10/945: {User interactive design; Environments; Toolboxes} - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06V 30/10: Character recognition - G06V 2201/10: Recognition assisted with metadata - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method, computer program product, and a system where a processor(s) obtains a video from a user, via a client, and segments the video into temporal shots that comprise a timeline of the video. The processor(s) cognitively analyze the video, by applying an image recognition algorithm to identify image entities in each temporal shot of the video and by applying a data structure comprising a user profile of the user to the temporal shots, to identity personal entities in each temporal shot of the video. The program code generates a search index for the video, utilizing the user entities (image entities and personal entities), where each entry of the search index is a given user entity and a linkage to a given temporal shot and the linkage indicates a location of the given user entity in the timeline of the video. ## Claims ### Claim 1 A computer-implemented method, comprising: obtaining, by one or more processors, from a user, via a client, a video to upload to a repository accessible to the one or more processors; segmenting, by the one or more processors, the video into temporal shots, wherein the temporal shots comprise a timeline of the video; cognitively analyzing, by the one or more processors, the video, by applying an image recognition algorithm, to the video, to identify image entities in each temporal shot of the video; cognitively analyzing, by the one or more processors, by applying a data structure comprising a user profile of the user to the temporal shots, to identity personal entities in each temporal shot of the video; and generating, by the one or more processors, a search index for the video, utilizing user entities, wherein the user entities comprise the image entities and the personal entities, wherein each entry of the search index comprises a given user entity, wherein the given user entity is selected from the user entities, and a linkage to a given temporal shot of the temporal shots, wherein the linkage indicates a location of the given user entity in the timeline of the video. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein applying the user profile comprises: monitoring, by the one or more processors, computing activities performed by the user, via the client, based on the client connecting, over a network, to one or more applications; analyzing, by the one or more processors, the computing activities performed by the user, in the one or more applications, to identify data comprising elements relevant to the user and relationships between the elements and the user; and generating, by the one or more processors, based on the analyzing, the data structure, wherein the data structure comprises the user profile. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein applying the user profile further comprises: converting, by the one or more processors, non-textual elements in the video to textual content, for each temporal shot of the temporal shots; and identifying, by the one or more processors, in the textual content of each temporal shot, the elements relevant to the user and the relationships between the elements and the user, wherein the elements comprise the personal entities. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: storing, by the one or more processors, the search index in an indexed repository. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising: obtaining, by the one or more processors, search parameters identifying one or more relevant user entities of the user entities in the search index; identifying, by the one or more processors, the relevant user entities; and searching, by the one or more processors, the video for the relevant user entities, wherein the searching comprises accessing the index repository to utilize the search index to locate the relevant user entities in the video. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising: formulating, responsive to the searching, by the one or more processors, search results, where the search results comprise the relevant user entities and for each relevant user entity, a location of the relevant user entity in the timeline, wherein the location comprises a start time and an end time. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein formulating the search results comprises ranking the search results based on relevance to the search parameters. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, further comprising: generating, by the one or more processors, a search deliverable, the generating comprising: obtaining, by the one or more processors, a portion of the temporal shots from the video, wherein each temporal shot of the portion comprises the location of the relevant user entity in the timeline for each user relevant entity; and assembling, by the one or more processors, the portion of the temporal shots into a new video. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, further comprising: providing, by the one or more processors, the search deliverable to the user, via the client. ### Claim 10 (depends on 8) The computer-implemented method of claim 8, wherein the assembling comprises assembling the portion of the temporal shots according to the ranking of the search results based on the relevance to the search parameters. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein the new video comprises more than one individual new videos, and where the providing of the search deliverable comprises providing links to each of the individual new videos. ### Claim 12 (depends on 5) The computer-implemented method of claim 5, wherein a format of the search parameters are selected from the group consisting of: text, voice, image, and video. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein applying the image recognition algorithm comprises accessing an image metadata repository accessible to the one or more processors. ### Claim 14 (depends on 3) The computer-implemented method of claim 3, wherein the non-textual elements comprise speech and audio, and wherein converting the elements comprises applying a speech to text processing algorithm to produce the textual content. ### Claim 15 (depends on 3) The computer-implemented method of claim 3, wherein the non-textual elements comprise embedded text in images comprising the video, wherein converting the elements comprises executing an optical character recognition process on the embedded text to convert the embedded text to the textual content, wherein the one or more applications comprise a social media website, and wherein the elements relevant to the user comprise images posted by the user on a social media website and tags associated with the images. ### Claim 16 (depends on 1) The computer-implemented method of claim 1, further comprising: prior to generating the search index, determining, by the one or more processors, a classification for the video, wherein obtaining the video from the user, via the client, further comprises obtaining the classification, from the user via the client; identifying, by the one or more processors, in the repository, another video uploaded by the user, wherein the classification of the other video is equivalent to the classification of the video; extracting, by the one or more processors, from a search index of the other video, user entities comprising the search index of the other video; searching, by the one or more processors, the video, for the user entities comprising the search index of the other video; and locating, by the one or more processors, a portion of the user entities comprising the search index of the other video in the video. ### Claim 17 (depends on 16) The computer-implemented method of claim 16, wherein the user entities further comprise the portion of the user entities. ### Claim 18 (depends on 1) The computer-implemented method of claim 1, further comprising: prior to generating the search index, generating, by the one or more processors, in a user interface of the client, an interface displaying the personal entities and respective linkages of the personal entities, wherein the interface comprises a point of entry by which the user can provide feedback; obtaining, by the one or more processors, the feedback from the user, provided via the interface; and updating, by the one or more processors, the user entities based on the feedback. ### Claim 19 A computer program product comprising: a computer readable storage medium readable by one or more processors and storing instructions for execution by the one or more processors for performing a method comprising: obtaining, by the one or more processors, from a user, via a client, a video to upload to a repository accessible to the one or more processors; segmenting, by the one or more processors, the video into temporal shots, wherein the temporal shots comprise a timeline of the video; cognitively analyzing, by the one or more processors, the video, by applying an image recognition algorithm, to the video, to identify image entities in each temporal shot of the video; cognitively analyzing, by the one or more processors, by applying a data structure comprising a user profile of the user to the temporal shots, to identity personal entities in each temporal shot of the video; and generating, by the one or more processors, a search index for the video, utilizing user entities, wherein the user entities comprise the image entities and the personal entities, wherein each entry of the search index comprises a given user entity, wherein the given user entity is selected from the user entities, and a linkage to a given temporal shot of the temporal shots, wherein the linkage indicates a location of the given user entity in the timeline of the video. ### Claim 20 A system comprising: a memory; one or more processors in communication with the memory; program instructions executable by the one or more processors via the memory to perform a method, the method comprising: obtaining, by the one or more processors, from a user, via a client, a video to upload to a repository accessible to the one or more processors; segmenting, by the one or more processors, the video into temporal shots, wherein the temporal shots comprise a timeline of the video; cognitively analyzing, by the one or more processors, the video, by applying an image recognition algorithm, to the video, to identify image entities in each temporal shot of the video; cognitively analyzing, by the one or more processors, by applying a data structure comprising a user profile of the user to the temporal shots, to identity personal entities in each temporal shot of the video; and generating, by the one or more processors, a search index for the video, utilizing user entities, wherein the user entities comprise the image entities and the personal entities, wherein each entry of the search index comprises a given user entity, wherein the given user entity is selected from the user entities, and a linkage to a given temporal shot of the temporal shots, wherein the linkage indicates a location of the given user entity in the timeline of the video.