# US 2020/0288255 A1 · Method and apparatus for sound object following > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0288255 A1 - Document ID: US20200288255A1 - Status: Published application - Kind: A1 - Application number: 16/812,199 - Filed: 2020-03-06 - Published: 2020-09-10 - Earliest claimed: 2019-03-08 - Assignees: LG Electronics Inc. - Applicants: LG Electronics Inc. - Inventors: Sungwon JUNG; Tacksung CHOI; Donghyun KANG; Seungsu LEE; Taegil CHO - Web page: https://patentgrep.com/patent/US20200288255A1 ## CPC - H04S 7/30: {Control circuits for electronic adaptation of the sound field} - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/253: {of extracted features} - G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries - G06V 10/806: {of extracted features} - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G10L 25/24: the extracted parameters being the cepstrum - G10L 25/57: for processing of video signals - H04R 29/001: {for loudspeakers (H04R29/007 takes precedence)} - H04R 29/008: {Visual indication of individual signal levels (visual indication of stereophonic sound image H04S7/40)} - H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)} - G06V 40/168: {Feature extraction; Face representation} - H04H 20/89: using three or more audio channels, e.g. triphonic or quadraphonic - H04H 20/95: characterised by a specific format, e.g. an encoded audio stream - H04R 2499/13: Acoustic transducers and sound field adaptation in vehicles - H04R 2499/15: Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops - H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - H04S: STEREOPHONIC SYSTEMS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N) ## Abstract The present disclosure relates to a method and apparatus for processing a multimedia signal. More specifically, the present disclosure relates to a method comprising obtaining a video frame and an audio frame from the multimedia signal; obtaining at least one video object from the video frame and at least one audio object from the audio frame; determining a correlation between the at least one video object and the at least one audio object; and performing a directional rendering on a specific audio object of the at least one audio object, based on a screen location of a specific video object related with the specific audio object from among the at least one video object according to the determined correlation, and an apparatus therefor. ## Claims ### Claim 1 A method of processing a multimedia signal by an apparatus, the method comprising: obtaining a video frame and an audio frame from the multimedia signal; obtaining at least one video object from the video frame and at least one audio object from the audio frame; determining a correlation between the at least one video object and the at least one audio object; and performing a directional rendering on a specific audio object of the at least one audio object, based on a screen location of a specific video object related with the specific audio object from among the at least one video object according to the determined correlation. ### Claim 2 (depends on 1) The method of claim 1, further comprising: performing a video classification on the at least one video object and an audio classification on the at least one audio object. ### Claim 3 (depends on 2) The method of claim 2, wherein each of the at least one audio object is classified into one of classes comprising a human speech, an animal sound, and a vehicle sound. ### Claim 4 (depends on 2) The method of claim 2, wherein each of the at least one video object is classified into one of classes comprising a human face, an animal, and a vehicle. ### Claim 5 (depends on 2) The method of claim 2, further comprising: obtaining feature information by performing a feature extraction on the at least one video object and the at least one audio object, wherein the correlation between the at least one video object and the at least one audio object is determined based on a result of the video classification, a result of the audio classification, the obtained feature information, and an object matching engine. ### Claim 6 (depends on 1) The method of claim 1, further comprising: obtaining feature information by performing a feature extraction on the at least one video object and the at least one audio object, wherein the correlation between the at least one video object and the at least one audio object is determined based on the obtained feature information and an object matching engine. ### Claim 7 (depends on 6) The method of claim 6, wherein the object matching engine comprises a model trained based on learning. ### Claim 8 (depends on 1) The method of claim 1, wherein performing the directional rendering on the specific audio object comprises: adjusting output gains of audio signals to be output to speakers based on the screen location of the specific video object related with the specific audio object. ### Claim 9 (depends on 1) The method of claim 1, wherein performing the directional rendering on the specific audio object comprises: adjusting output rates of audio signals related to speakers based on the screen location of the specific video object related with the specific audio object. ### Claim 10 (depends on 9) The method of claim 9, wherein adjusting the output rates of the audio signals comprises: adjusting the output rates of the audio signals related to the speakers based on a horizontal spacing between the speakers, a screen width, and the screen location of the specific video object. ### Claim 11 (depends on 10) The method of claim 10, wherein, based on the speakers comprising a left speaker and a right speaker, the output rates of the audio signals are adjusted such that a sum of an output rate of an audio signal related to the left speaker and an output rate of an audio signal related to the right speaker is equal to a constant value. ### Claim 12 (depends on 11) The method of claim 11, wherein the constant value is 1. ### Claim 13 (depends on 10) The method of claim 10, wherein, based on the speakers comprising a top left speaker, a top right speaker, a bottom left speaker, and a bottom right speaker, the output rates of the audio signals are adjusted such that a sum of an output rate of an audio signal related to the top left speaker, an output rate of an audio signal related to the top right speaker, an output rate of an audio signal related to the bottom left speaker, and an output rate of an audio signal related to the bottom right speaker is equal to a constant value. ### Claim 14 (depends on 13) The method of claim 13, wherein the constant value is 1. ### Claim 15 An apparatus configured to process a multimedia signal, the apparatus comprising: a memory storing instructions; and at least one processor operatively coupled to the memory and configured to, when executing the instructions, implement operations comprising: obtaining a video frame and an audio frame from the multimedia signal; obtaining at least one video object from the video frame and at least one audio object from the audio frame; determining a correlation between the at least one video object and the at least one audio object; and performing a directional rendering on a specific audio object of the at least one audio object, based on a screen location of a specific video object related with the specific audio object from among the at least one video object according to the determined correlation. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the operations further comprise: performing a video classification on the at least one video object and an audio classification on the at least one audio object. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein each of the at least one audio object is classified into one of classes comprising a human speech, an animal sound, and a vehicle sound. ### Claim 18 (depends on 16) The apparatus of claim 16, wherein each of the at least one video object is classified into one of classes comprising a human face, an animal, and a vehicle. ### Claim 19 (depends on 16) The apparatus of claim 16, wherein the operations further comprise: obtaining feature information by performing a feature extraction on the at least one video object and the at least one audio object, wherein the correlation between the at least one video object and the at least one audio object is determined based on a result of the video classification, a result of the audio classification, the obtained feature information, and an object matching engine. ### Claim 20 (depends on 15) The apparatus of claim 15, wherein the operations further comprise: obtaining feature information by performing a feature extraction on the at least one video object and the at least one audio object, wherein the correlation between the at least one video object and the at least one audio object is determined based on the obtained feature information and an object matching engine.