# US 10,482,902 B2 · Method and system to evaluate and quantify user-experience (UX) feedback

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,902 B2
- Document ID: US10482902B2
- Status: Granted
- Kind: B2
- Application number: 15/942,782
- Filed: 2018-04-02
- Published: 2018-10-04
- Granted: 2019-11-19
- Earliest claimed: 2017-03-31
- Assignees: Martin Benjamin Seider
- Applicants: Martin Benjamin Seider
- Inventors: Martin Benjamin Seider
- Web page: https://patentgrep.com/patent/US10482902B2

## CPC

- G10L 25/60: for measuring the quality of voice signals
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G10L 25/30: using neural networks
- G10L 25/48: specially adapted for particular use
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The exemplified methods and systems facilitate the evaluation of user experience feedback in a manner that is secure and private by transmitting only representation of a user's response over the network (rather than the original data files of the recording). That is, complex interaction associated with a user to a stimuli is recorded, evaluated, and condensed through a machine learning operation (specifically, a convolutional neural network) performed at the user's computing device. Indeed, only a representation of the interaction is exposed over the network when it is transmitted for subsequent action there-at. Specific audio or image recording used to evaluate the user's user-experience feedback is maintained at the user's computing device. In some embodiments, the recording is preprocessed to generate a series of matrices that can be directly fed into the machine learning operation.

## Claims

### Claim 1

A computer-implemented method to evaluate and quantify user-experience (UX) feedback in response to a user-interface presentation (e.g., a themed ringtone), the method comprising:
generating, by a processor of a computing device, via one or more speakers of, or connected to, the computing device, a multimedia output of a plurality of selectable multimedia outputs accessible from the computing device, wherein the multimedia output comprises an audio output associated with a stored audio file, and wherein the multimedia output is generated upon receipt of an electronic communication or notification at the computing device;
recording, by the processor, via one or more microphones of, or connected to, the computing device, an audio stream as part of a user-experience (UX) feedback to the user-interface presentation, wherein the audio stream is recorded immediately following, and/or during, the audio output being generated, and wherein the audio stream is recorded as an audio file or as a cached data;
determining, by the processor, a spectrum profile of each, or a substantial portion, of a plurality of audio segments of the recorded audio file or the cached data; and
determining, by the processor, a matrix comprising the plurality of spectrum profiles for each of the plurality of audio segments and providing the matrix to a machine learning operation configured to analyze the user-experience (UX) feedback, wherein the matrix is used as an input for the machine learning operation configured with weights specific to the multimedia output to evaluate and quantify user-experience (UX) feedback of the multimedia output.

### Claim 2 (depends on 1)

The method of claim 1, wherein the machine learning operation comprises a convolutional neural network, and wherein the inputted matrix is arranged as a two-dimensional matrix.

### Claim 3 (depends on 2)

The method of claim 2, wherein the inputted matrix is determined by:
segmenting, by the processor, the recorded audio file or the cached into a plurality of segments;
normalizing, by the processor, the plurality of segments;
determining, by the processor, intensity of frequencies bands of the plurality of segments by dividing into frequency components of the plurality of segments; and
generating, by the processor, the two-dimensional matrix, wherein the two-dimensional matrix is of dimension m*n, wherein m is a number of segments of the plurality of segments and n is a number of frequency bands of a plurality of frequency bands to which the segment is evaluated, and wherein each matrix element has at least a scalar value corresponding to an intensity of a given frequency band for a given segment.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
acquiring, by the processor, via one or more accelerometers of the computing device, one or more detected accelerometer signals immediately following the audio output as an accelerometer file or as cached data, wherein the one or more detected accelerometer signals comprise an additional part of the user-experience (UX) feedback of the user-interface presentation; and
determining by the processor, a second matrix comprising the plurality of accelerometer signals and providing the second matrix as an input to the machine learning operation.

### Claim 5 (depends on 1)

The method of claim 1, wherein the plurality of selectable multimedia outputs further comprises a visualization element (e.g., ringtone includes associated ringtone graphics), wherein the visualization element is rendered and presented on a display of the computing device contemporaneously with the audio output.

### Claim 6 (depends on 5)

The method of claim 5, wherein the plurality of selectable multimedia outputs further comprises a haptics element (e.g., ringtone includes associated vibration pattern), wherein the haptics element triggers rotation of a motor of the computing device to produce vibration of the computing device contemporaneously with the audio output.

### Claim 7 (depends on 1)

The method of claim 1, wherein the computing device is selected from the group consisting of smartphone, wearable devices, smartwatches, smart glasses, laptops, and desktops.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
wherein the output of the machine learning operation is used to assess a positive user-experience and/or a negative user-experience of the user-interface presentation.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
transmitting, by the processor, over a network, output of the machine learning operation to an external UX-analysis service, wherein the output of the machine learning operation is used by the UX-analysis service to trigger a reward to an account associated with the user.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
encrypting, by the processor, output of the machine learning operation prior to transmission of the output to the external UX-analysis service.

### Claim 11

A system comprising:
a processor; and
a memory having instructions stored thereon, wherein execution of the instructions by the processor, cause the processor to:
generate, via one or more speakers of, or connected to, the system, a multimedia output of a plurality of selectable multimedia outputs accessible to the system, wherein the multimedia output comprises an audio output (e.g., a ringtone) associated with a stored audio file, and wherein the multimedia output is generated upon receipt of an electronic communication or notification (e.g., a SMS message, electronic mail) at the system;
record, via one or more microphones of, or connected to, the system, an audio stream as part of a user-experience (UX) feedback of a user-interface presentation, wherein the audio stream is recorded immediately following, and/or during, the audio output being generated, and wherein the audio stream is recorded as an audio file or as a cached data;
determine, a spectrum profile of each, or a substantial portion, of a plurality of audio segments of the recorded audio file or the cached data; and
determine a matrix comprising the plurality of spectrum profiles for each of the plurality of audio segments and provide the matrix to a machine learning artifact configured to analyze the user-experience (UX) feedback, wherein the matrix is used as an input to the machine learning artifact configured with weights specific to the multimedia output to evaluate and quantify a user-experience (UX) feedback of the multimedia output.

### Claim 12 (depends on 11)

The system of claim 11, wherein the machine learning artifact comprises a convolutional neural network, and wherein the inputted matrix is arranged as a two-dimensional matrix.

### Claim 13 (depends on 12)

The system of claim 12, wherein the instructions, when executed by the processor, further cause the processor to:
segment the recorded audio file or the cached into a plurality of segments;
normalize the plurality of segments; and
determine intensity of the frequencies bands of the plurality of segments by dividing into frequency components of the plurality of segments; and
generate the two-dimensional matrix, wherein the two-dimensional matrix is of dimension m*n, wherein m is a number of segments of the plurality of segments and n is a number of frequency bands of the plurality of frequency bands, and each matrix element has at least a scalar value corresponding to an intensity of a given frequency band for a given segment.

### Claim 14 (depends on 11)

The system of claim 11, wherein the instructions, when executed by the processor, further cause the processor to:
encrypt output of the machine learning artifact; and
transmit, over a network, output of the machine learning artifact to an external UX-analysis service, wherein the output of the machine learning artifact is used by the external UX-analysis service to trigger a reward to an account associated with the user.

### Claim 15 (depends on 11)

The system of claim 11, wherein the instructions when executed by the processor, further cause the processor to:
acquire, via one or more accelerometers of the system, one or more detected accelerometer signals immediately following or during the audio output as an accelerometer file or as cached data, wherein the one or more detected accelerometer signals comprise an additional part of the user-experience (UX) feedback of the user-interface presentation; and
determine a second matrix comprising the plurality of accelerometer signals and provide the second matrix to the UX-analysis computing device, wherein the second matrix is used as a second input to the machine learning artifacts.

### Claim 16

A non-transitory computer readable medium having instructions stored thereon, wherein execution of the instructions by a processor of a computing device causes the processor to:
generate, via one or more speakers of, or connected to, the computing device, a multimedia output of a plurality of selectable multimedia outputs accessible to the computing device, wherein the multimedia output comprises an audio output (e.g., a ringtone) associated with a stored audio file, and wherein the multimedia output is generated upon receipt of an electronic communication or notification (e.g., a SMS message, electronic mail) at the computing device;
record, via one or more microphones of, or connected to, the computing device, an audio stream immediately following, and/or during, the audio output as part of a user-experience (UX) feedback of the user-interface presentation, wherein the audio stream is recorded as an audio file or as a cached data;
determine, a spectrum profile of each, or a substantial portion, of a plurality of audio segments of the recorded audio file or the cached data; and
determine a matrix comprising the plurality of spectrum profiles for each of the plurality of audio segments and provide the matrix to a machine learning artifact configured to analyze the user-experience (UX) feedback, wherein the matrix is used as an input to a machine learning artifact configured with weights specific to the multimedia output to evaluate and quantify a user-experience (UX) feedback to the multimedia output.

### Claim 17 (depends on 16)

The computer readable medium of claim 16, wherein the machine learning artifact comprises a convolutional neural network, and wherein the inputted matrix is arranged as a two-dimensional matrix.

### Claim 18 (depends on 16)

The computer readable medium of claim 16, wherein the instructions, when executed by the processor, further cause the processor to:
segment the recorded audio file or the cached data into a plurality of segments;
normalize the plurality of segments; and
determine intensity of the frequencies bands of the plurality of segments by dividing into frequency components of the plurality of segments; and
generate the two-dimensional matrix, wherein the two-dimensional matrix is of dimension m*n, wherein m is a number of segments of the plurality of segments and n is a number of frequency bands of the plurality of frequency bands, and each matrix element has at least a scalar value corresponding to an intensity of a given frequency band for a given segment.

### Claim 19 (depends on 16)

The computer readable medium of claim 16, wherein the instructions, when executed by the processor, further cause the processor to:
encrypt output of the machine learning artifact; and
transmit, over a network, output of the machine learning artifact to an external UX analysis service, wherein the output of the machine learning artifact is used by the external UX analysis service to trigger a reward to an account associated with the user.

### Claim 20 (depends on 16)

The computer-readable medium of claim 16, wherein the instructions when executed by the processor, further cause the processor to:
acquire, via one or more accelerometers of the computing device, one or more detected accelerometer signals immediately following and/or during the audio output as an accelerometer file or as cached data, wherein the one or more detected accelerometer signals comprise an additional part of the user-experience (UX) feedback of the user-interface presentation; and
determine a second matrix comprising the plurality of acquired accelerometer signals and providing the second matrix to the machine learning artifact, wherein the second matrix is used as a second input to the machine learning artifact configured with weights specific to the multimedia output to evaluate and quantify a user-experience (UX) feedback to the multimedia output.