# US 2025/0147720 A1 · System and method for generating audio during traversing of a user interface

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0147720 A1
- Document ID: US20250147720A1
- Status: Published application
- Kind: A1
- Application number: 18/778,108
- Filed: 2024-07-19
- Published: 2025-05-08
- Earliest claimed: 2023-11-03
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Sumit PANWAR; Avinash SINGH
- Web page: https://patentgrep.com/patent/US20250147720A1

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 25/21: the extracted parameters being power information
- H04R 2430/01: Aspects of volume control, not necessarily automatic, in sound systems
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

A method for generating an audio signal includes capturing, during a traversal event, at least one audio stream associated with one or more of a content, an ambient audio stream, and at least one user interface (UI) frame; obtaining a set of audio features, and obtaining one or more of UI elements; determining an audio gain based on a prioritization and a weighted average of the extracted set of audio features; determining a UI context based on a UI contextual score of the extracted one or more UI elements; obtaining an audio map comprising at least one biased audio file and at least one unbiased audio file; and obtaining the audio signal based on a priority order assigned to the at least one biased audio file and the at least one unbiased audio file associated with the audio map.

## Claims

### Claim 1

A method for obtaining an audio signal during a traversal event associated with a user interface (UI), the method comprising:
capturing, during the traversal event indicative of performing navigation on the UI, at least one audio stream associated with content, an ambient audio stream, and at least one UI frame;
obtaining a set of audio features based on the at least one audio stream and the ambient audio stream, and obtaining one or more UI elements from a set of images generated based on the at least one UI frame;
determining an audio gain based on a prioritization and a weighted average of the set of audio features, wherein the audio gain indicates an intensity of the at least one audio associated with the UI;
determining a UI context based on a UI contextual score of the one or more UI elements, wherein the UI context indicates context of activities during the traversal event in the UI;
obtaining an audio map comprising at least one biased audio file and at least one unbiased audio file, wherein the at least one biased audio file is determined based on the set of audio features and the at least one unbiased audio file is determined based on the UI context; and
obtaining the audio signal based on a priority order assigned to the at least one biased audio file and the at least one unbiased audio file associated with the audio map, the set of audio features, and the audio gain, such that the audio signal is rendered during the traversal event on the UI.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising rendering the audio signal via at least one audio channel on a user device corresponding to the UI.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the set of audio features corresponds to a content gain, a content frequency, and a duration of the at least one audio, based on the at least one audio stream being associated with the content, and
wherein the set of audio features corresponds to an ambience gain and a vocal gain of the at least one audio, based on the at least one audio being associated with the ambient audio stream.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the one or more UI elements indicate interactive components of the UI, and
wherein the interactive components comprise at least one of a text, an object, a hierarchal tree, a logo, and an icon.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the determining the audio gain based on the prioritization and the weighted average of the set of audio features comprises:
obtaining the set of audio features associated with the at least one audio stream and the ambient audio stream;
obtaining a user voice input;
determining a gain value corresponding to the set of audio features;
determining the prioritization of the set of audio features based on a prioritized audio list and the gain value;
obtaining a weight associated with each of the set of audio features based on a pre-stored weight values;
determining the weighted average corresponding to the set of audio features based on a smoothing factor and the weight associated with each audio feature of the set of audio features; and
determining the audio gain based on the prioritization and the weighted average such that the change in volume between the audio signal to be generated for the UI and the at least one audio associated with the content being played in the background, is performed at a rate below a change rate threshold.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the determining the UI context based on the UI contextual score of the one or more UI elements comprises:
performing an element filtering based on the one or more UI elements, wherein the element filtering comprises identifying each of the one or more UI elements separated based on respective positions within the at least one UI frame;
determining a bias score associated with each of the one or more UI elements, wherein the bias score indicates a probability of each of the one or more UI elements matching with a context corresponding to the UI;
determining a contextual classification of an application corresponding to the UI, wherein the contextual classification indicates a category of the application determined using a pre-trained language machine learning model;
obtaining the UI contextual score based on the bias score and the contextual classification for each of the one or more UI elements; and
determining the UI context based on the UI contextual score.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the determining the at least one biased audio file based on the set of audio features comprises:
scaling a pre-generated audio based on the audio gain; and
modulating the pre-generated audio based on the set of audio features to determine the at least one biased audio file, wherein the at least one biased audio file aligns with the UI context.

### Claim 8 (depends on 1)

The method according to claim 1, wherein determining the at least one unbiased audio file based on the UI context comprises:
obtaining the UI contextual score for each of the one or more UI elements; and
determining the at least one unbiased audio file using a recurring neural network based on the at least one audio stream associated with the content and a corresponding contextual audio indicating a semantic content of the at least one audio.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising:
aggregating the at least one biased audio file and the at least one unbiased audio file to obtain the audio map, wherein the audio map indicates a sequence of the at least one biased audio file and the at least one unbiased audio file along with associated audio-type scores.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the obtaining the audio signal comprises:
obtaining the priority order of the audio map, the set of audio features, and the audio gain;
generating one or more contextual audio files using a diffusion model based on the priority order of the audio map, wherein the one or more contextual audio files indicate semantic content of the audio map;
generating a context of background audio based on a modulation of the audio map with the one or more contextual audio files;
generating one or more mixed audio files based on mixing the at least one biased audio file of the audio map with the at least one unbiased audio file of the audio map; and
obtaining the audio signal based on the one or more mixed audio files and the audio gain.

### Claim 11 (depends on 1)

The method according to claim 1, further comprising:
capturing, while a user device corresponding to the UI is operated, the at least one audio stream associated with a physical event and the ambient audio stream for the physical event;
extracting the set of audio features based on the at least one audio stream and the ambient audio stream;
determining the audio gain based on the prioritization and the weighted average of the set of audio features;
determining an event context based on the physical event; and
obtaining the audio signal based on the event context and the audio gain such that the generated audio signal corresponds to the physical event.

### Claim 12

An electronic device for obtaining an audio signal during a traversal event associated with a user interface (UI), the electronic device comprising:
a memory storing one or more instructions; and
one or more processors operatively coupled to the memory and configured to execute the one or more instructions,
wherein of the one or more instructions, when executed by the one or more processors, cause the electronic device to:
capture, during the traversal event, at least one audio stream associated with content, an ambient audio stream, and at least one UI frame, wherein the traversal event is indicative of performing navigation on the UI;
obtain a set of audio features based on the at least one audio stream and the ambient audio stream, and extract one or more UI elements from a set of images generated based on the at least one UI frame;
determine an audio gain based on a prioritization and a weighted average of the set of audio features, wherein the audio gain indicates an intensity of the at least one audio associated with the UI;
determine a UI context based on a UI contextual score of the one or more UI elements, wherein the UI context indicates context of activities during the traversal event in the UI;
obtain an audio map comprising at least one biased audio file and at least one unbiased audio file, wherein the at least one biased audio file is determined based on the set of audio features and the at least one unbiased audio file is determined based on the UI context; and
obtain the audio signal based on a priority order assigned to the at least one biased audio file and the at least one unbiased audio file associated with the audio map, the set of audio features, and the audio gain such that the audio signal is rendered during the traversal event on the UI.

### Claim 13 (depends on 12)

The electronic device according to claim 12, wherein the one or more instructions, when executed by the one or more processors, further cause the electronic device to:
render the audio signal via at least one audio channel on a user device corresponding to the UI.

### Claim 14 (depends on 12)

The electronic device according to claim 12, wherein the set of audio features corresponds to a content gain, a content frequency, and a duration of the at least one audio, based on the at least one audio stream being associated with the content; and
wherein the set of audio features corresponds to an ambience gain and a vocal gain of the at least one audio, based on the at least one audio being associated with the ambient audio stream.

### Claim 15 (depends on 12)

The electronic device according to claim 12, wherein the one or more UI elements indicate interactive components of the UI, and
wherein the interactive components include at least one of a text, an object, a hierarchal tree, a logo, and an icon.

### Claim 16 (depends on 12)

The electronic device according to claim 12, wherein the one or more instructions, when executed by the one or more processors, further cause the electronic device to:
obtain the set of audio features associated with the at least one audio stream and the ambient audio stream;
obtain a user voice input;
determine a gain value corresponding to the set of audio features;
determine the prioritization of the set of audio features based on a prioritized audio list and the gain value;
obtain a weight associated with each of the set of audio features based on a pre-stored weight values;
determine the weighted average corresponding to the set of audio features based on a smoothing factor and the weight associated with each of the set of audio features; and
determine the audio gain based on the prioritization and the weighted average such that the change in volume between the audio signal to be generated for the UI and the at least one audio associated with the content being played in the background is performed at a rate below a change rate threshold.

### Claim 17 (depends on 12)

The electronic device according to claim 12, wherein the one or more instructions, when executed by the one or more processors, further cause the electronic device to:
perform an element filtering based on the one or more UI elements, wherein the element filtering comprises identifying each of the one or more UI elements separated based on respective positions within the at least one UI frame;
determine a bias score associated with each of the one or more UI elements, wherein the bias score indicates a probability of each of the one or more UI elements matching with a context corresponding to the UI;
determine a contextual classification of an application corresponding to the UI, wherein the contextual classification indicates a category of the application determined using a pre-trained language machine learning model;
obtain the UI contextual score based on the bias score and the contextual classification for each of the one or more UI elements; and
determine the UI context based on the UI contextual score.

### Claim 18 (depends on 12)

The electronic device according to claim 12, wherein the one or more instructions, when executed by the one or more processors, further cause the electronic device to:
scale a pre-generated audio based on the audio gain; and
modulate the pre-generated audio based on the set of audio features to determine the at least one biased audio file, wherein the at least one biased audio file aligns with the UI context.

### Claim 19 (depends on 12)

The electronic device according to claim 12, wherein the one or more instructions, when executed by the one or more processors, further cause the electronic device to:
obtain the UI contextual score for each of the one or more UI elements; and
determine the at least one unbiased audio file using a recurring neural network based on the at least one audio stream associated with the content and a corresponding contextual audio indicating a semantic content of the at least one audio.

### Claim 20

A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to:
capture, during the traversal event indicative of performing navigation on the UI, at least one audio stream associated with content, an ambient audio stream, and at least one UI frame;
obtain a set of audio features based on the at least one audio stream and the ambient audio stream, and obtaining one or more UI elements from a set of images generated based on the at least one UI frame;
determine an audio gain based on a prioritization and a weighted average of the set of audio features, wherein the audio gain indicates an intensity of the at least one audio associated with the UI;
determine a UI context based on a UI contextual score of the one or more UI elements, wherein the UI context indicates context of activities during the traversal event in the UI;
obtain an audio map comprising at least one biased audio file and at least one unbiased audio file, wherein the at least one biased audio file is determined based on the set of audio features and the at least one unbiased audio file is determined based on the UI context; and
obtain the audio signal based on a priority order assigned to the at least one biased audio file and the at least one unbiased audio file associated with the audio map, the set of audio features, and the audio gain, such that the generated audio signal is rendered during the traversal event on the UI.