# US 2020/0288256 A1 · Method and apparatus for sound object following > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0288256 A1 - Document ID: US20200288256A1 - Status: Published application - Kind: A1 - Application number: 16/812,183 - Filed: 2020-03-06 - Published: 2020-09-10 - Earliest claimed: 2019-03-08 - Assignees: LG Electronics Inc. - Applicants: LG Electronics Inc. - Inventors: Sungwon JUNG; Tacksung CHOI; Donghyun KANG; Seungsu LEE; Taegil CHO - Web page: https://patentgrep.com/patent/US20200288256A1 ## CPC - H04S 7/30: {Control circuits for electronic adaptation of the sound field} - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/253: {of extracted features} - G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries - G06V 10/806: {of extracted features} - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G10L 25/24: the extracted parameters being the cepstrum - G10L 25/57: for processing of video signals - H04R 29/001: {for loudspeakers (H04R29/007 takes precedence)} - H04R 29/008: {Visual indication of individual signal levels (visual indication of stereophonic sound image H04S7/40)} - H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)} - G06V 40/168: {Feature extraction; Face representation} - H04H 20/89: using three or more audio channels, e.g. triphonic or quadraphonic - H04H 20/95: characterised by a specific format, e.g. an encoded audio stream - H04R 2499/13: Acoustic transducers and sound field adaptation in vehicles - H04R 2499/15: Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops - H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - H04S: STEREOPHONIC SYSTEMS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N) ## Abstract The present disclosure relates to a method and apparatus for processing a multimedia signal. More specifically, the present disclosure relates to a method comprising obtaining at least one video object from the multimedia signal and at least one audio object from the multimedia signal, extracting video feature information for the at least one video object and audio feature information for the at least one audio object, and determining a correlation between the at least one video object and the at least one audio object through an object matching engine based on the video feature information and the audio feature information, and an apparatus therefor. ## Claims ### Claim 1 A method of processing a multimedia signal by an apparatus, the method comprising: obtaining at least one video object from the multimedia signal and at least one audio object from the multimedia signal; extracting video feature information for the at least one video object and audio feature information for the at least one audio object; and determining a correlation between the at least one video object and the at least one audio object through an object matching engine based on the video feature information and the audio feature information. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the correlation between the at least one video object and the at least one audio object comprises: obtaining information about a relationship between each of the at least one video object and a specific audio object of the at least one audio object through the object matching engine based on the video feature information and the audio feature information; and determining, from among the at least one video object, a specific video object related to the specific audio object based on the information about the relationship between each of the at least one video object and the specific audio object. ### Claim 3 (depends on 2) The method of claim 2, wherein determining the specific video object related to the specific audio object comprises: based on a value of the information about the relationship being greater than a predetermined value, determining a video object related to the value of the information about the relationship as the specific video object. ### Claim 4 (depends on 3) The method of claim 3, wherein determining the specific video object related to the specific audio object further comprises: based on the least one video object comprising a plurality of video objects, determining, from among the plurality of video objects, a video object related to a greatest value of the information about the relationship as the specific video object. ### Claim 5 (depends on 2) The method of claim 2, wherein determining the specific video object related to the specific audio object comprises: based on a value of the information about the relationship being less than a predetermined value, determining a video object related to the value of the information about the relationship as the specific video object. ### Claim 6 (depends on 5) The method of claim 5, wherein determining the specific video object related to the specific audio object further comprises: based on the least one video object comprising a plurality of video objects, determining, from the plurality of video objects, a video object related to a smallest value of the information about the relationship as the specific video object. ### Claim 7 (depends on 2) The method of claim 2, wherein the information about the relationship has a real number value. ### Claim 8 (depends on 1) The method of claim 1, wherein the video feature information is extracted based on a vertical length and a horizontal length of a lip skeleton. ### Claim 9 (depends on 1) The method of claim 1, wherein the video feature information is extracted based on: a ratio between a distance between a top boundary of a first rectangular region including a face and a top boundary of a second rectangular region including lips within the first rectangular region, a height of the second rectangular region, and a distance between a bottom boundary of the second rectangular region and a bottom boundary of the first rectangular region; and a ratio between a distance between a left boundary of the second rectangular region and a left boundary of the first rectangular region, a width of the second rectangular region, and a distance between a right boundary of the second rectangular region and a right boundary of the first rectangular region. ### Claim 10 (depends on 1) The method of claim 1, wherein the audio feature information is extracted based on linear prediction coding (LPC). ### Claim 11 (depends on 1) The method of claim 1, wherein the audio feature information is extracted based on a log-Mel filters-of-bank. ### Claim 12 (depends on 1) The method of claim 1, wherein the audio feature information is extracted based on Mel-frequency cepstral coefficients (MFCC). ### Claim 13 (depends on 1) The method of claim 1, wherein the audio feature information comprises onset information about the at least one audio object. ### Claim 14 (depends on 1) The method of claim 1, wherein the object matching engine comprises a model trained based on learning. ### Claim 15 An apparatus configured to process a multimedia signal, the apparatus comprising: a memory storing instructions; and at least one processor operatively coupled to the memory and configured to, when executing the instructions, implement operations comprising: obtaining at least one video object from the multimedia signal and at least one audio object from the multimedia signal; extracting video feature information for the at least one video object and audio feature information for the at least one audio object; and determining a correlation between the at least one video object and the at least one audio object through an object matching engine based on the video feature information and the audio feature information. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein determining the correlation between the at least one video object and the at least one audio object comprises: obtaining information about a relationship between each of the at least one video object and a specific audio object of the at least one audio object through the object matching engine based on the video feature information and the audio feature information; and determining, from among the at least one video object, a specific video object related to the specific audio object based on the information about the relationship between each of the at least one video object and the specific audio object. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein determining the specific video object related to the specific audio object comprises: based on a value of the information about the relationship being greater than a predetermined value, determining a video object related to the value of the information about the relationship as the specific video object. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein determining the specific video object related to the specific audio object further comprises: based on the least one video object comprising a plurality of video objects, determining, from among the plurality of video objects, a video object related to a greatest value of the information about the relationship as the specific video object. ### Claim 19 (depends on 16) The apparatus of claim 16, wherein determining the specific video object related to the specific audio object comprises: based on a value of the information about the relationship being less than a predetermined value, determining a video object related to the value of the information about the relationship as the specific video object. ### Claim 20 (depends on 19) The apparatus of claim 19, wherein determining the specific video object related to the specific audio object further comprises: based on the least one video object comprising a plurality of video objects, determining, from the plurality of video objects, a video object related to a smallest value of the information about the relationship as the specific video object.