# US 2019/0379342 A1 · Systems and methods using predictive analysis of audio signal to generate haptic data for enhanced user experience

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0379342 A1
- Document ID: US20190379342A1
- Status: Published application
- Kind: A1
- Application number: 16/435,344
- Filed: 2019-06-07
- Published: 2019-12-12
- Earliest claimed: 2018-06-07
- Assignees: Lofelt GmbH
- Applicants: Lofelt GmbH
- Inventors: Maximilian Weber; Daniel Büttner; Sebastian Roos; James Mazur
- Web page: https://patentgrep.com/patent/US20190379342A1

## CPC

- G06N 7/02: using fuzzy logic (computing arrangements based on biological models G06N3/00; computing arrangements using knowledge-based models G06N5/00)
- G08B 6/00: Tactile signalling systems, e.g. tactile personal calling systems
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/065: Analogue means
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 20/20: Ensemble learning
- G10H 1/0091: {Means for obtaining special acoustic effects (combined with modulation G10H1/043)}
- G10H 1/12: by filtering complex waveforms (G10H1/14, G10H1/16 take precedence)
- G10H 1/46: Volume control
- H03G 3/3005: {in amplifiers suitable for low-frequencies, e.g. audio amplifiers (H03G3/32, H03G3/34 take precedence)}
- H03G 9/14: for gain control and tone control
- H04R 1/1008: {Earpieces of the supra-aural or circum-aural type}
- H04R 1/1075: {Mountings of transducers in earphones or headphones}
- H04R 5/033: Headphones for stereophonic communication {(details thereof, e.g. relating to batteries, cables or control elements H04R1/10)}
- H04R 5/04: Circuit arrangements, {e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments (combinations of amplifiers H03F3/68; stereophonic systems H04S)}
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G10H 2210/281: Reverberation or echo
- G10H 2210/315: Dynamic effects for musical purposes, i.e. musical sound effects controlled by the amplitude of the time domain audio envelope, e.g. loudness-dependent tone colour or musically desired dynamic range compression or expansion
- G10H 2220/311: with controlled tactile or haptic feedback effect; output interfaces therefor
- G10H 2250/025: Envelope processing of music signals in, e.g. time domain, transform domain or cepstrum domain
- H03G 5/165: {Equalizers; Volume or gain control in limited frequency bands}
- H04R 3/04: for correcting frequency response
- H04R 2205/022: Plurality of transducers corresponding to a plurality of sound channels in each earpiece of headphones or in a single enclosure
- H04R 2400/03: Transducers capable of generating both sound as well as tactile vibration, e.g. as used in cellular phones
- H04R 2430/03: Synergistic effects of band splitting and sub-band processing
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- H03G: CONTROL OF AMPLIFICATION
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

Systems and methods for generating a haptic output from an audio signal having a continuous stream of sampled digital audio data are provided. A haptic processing system receives the digital audio data, analyses the digital audio data for processing and extracts haptic signals for generating a haptic effect through an actuator. The method includes passing the digital audio signal on through dynamic processor(s), adjusting the dynamic range of the digital audio signal, extracting the signal envelope of the audio data, synthesising low-frequency signals from the extracted signal envelope, and enhancing the low-frequency content using a resonator. The haptic output is generated by mixing the digital audio signal with outputs from the different modules of the haptic processing system. An analytics module monitors, controls and adjusts the processing of the digital audio signal at the noise gate module, the compressor module and the envelope module to enhance the haptic output.

## Claims

### Claim 1

A method of processing haptic data from an input audio signal to produce haptic output, the method comprising:
receiving an audio input signal simultaneously in a digital signal processing chain and an analytics processor;
processing the input audio signal in the digital signal processing chain having at least one dynamic processor;
processing the input audio signal in the analytics processor by analyzing the input audio characteristics to select at least one predictive algorithm and a dataset for training the predictive algorithm; and
using the selected predictive algorithm to calculate at least one control parameter to control the at least one dynamic processor in the digital signal processing chain.

### Claim 2 (depends on 1)

The method of processing haptic data of claim 1, wherein the predictive algorithm is trained using the real-time data.

### Claim 3 (depends on 2)

The method of processing haptic data of claim 2, wherein the real-time data comprises of RMS values and peak values and a sequence of steps to determine gain and/or threshold.

### Claim 4 (depends on 1)

The method of processing haptic data of claim 1, wherein the predictive algorithm is trained using both the pre-stored data and the real-time data.

### Claim 5 (depends on 1)

The method of processing haptic data of claim 1, wherein the predictive algorithm implements at least one supervised learning algorithm.

### Claim 6 (depends on 1)

The method of processing haptic data of claim 1, wherein the predictive algorithm implements at least one unsupervised learning algorithm.

### Claim 7 (depends on 1)

The method of processing haptic data of claim 1, wherein the at least one control parameter is an amplitude.

### Claim 8 (depends on 1)

The method of processing haptic data of claim 1, wherein the at least one control parameter is a gain.

### Claim 9 (depends on 1)

The method of processing haptic data of claim 1, wherein the at least one control parameter is a threshold.

### Claim 10

A method of processing haptic data from an input audio signal to produce haptic output, the method comprising:
receiving an audio input signal simultaneously in a digital signal processing chain and an analytics processor;
processing the input audio signal in the digital signal processing chain having at least one dynamic processor;
processing the input audio signal in the analytics processor by analyzing the input audio signal characteristics to select at least one predictive algorithm and a dataset for training the predictive algorithm;
using the selected predictive algorithm to determine the input audio signal characteristics to predict haptic data; and
producing the haptic output using one or more actuators.

### Claim 11

A haptic processing system for processing a haptic experience from an input audio signal, the haptic processing system comprising:
a digital signal processing chain with one or more dynamic processors;
an analytics processor comprising a predictive and intelligence processor and memory, the memory divided into one or more segments to store waveform analysis parameters, haptic data, pre-stored data, and one or more predictive algorithms, wherein the one or more predictive algorithms are trained using the pre-stored data;
a predictive and intelligence processor to predict haptic output based at least on the input audio signal characteristics in real time and to produce the haptic output in one or more haptic actuators.

### Claim 12 (depends on 11)

The haptic processing system of claim 11, wherein the one or more predictive algorithms are trained using the real-time data.

### Claim 13 (depends on 12)

The haptic processing system of claim 12, wherein the real-time data comprises of RMS values and peak values and a sequence of steps to determine gain and/or threshold.

### Claim 14 (depends on 11)

The haptic processing system of claim 11, wherein the one or more predictive algorithms are trained using both the pre-stored data and the real-time data.

### Claim 15 (depends on 11)

The haptic processing system of claim 11, wherein the one or more predictive algorithms implement at least one supervised learning algorithm.

### Claim 16 (depends on 11)

The haptic processing system of claim 11, wherein the one or more predictive algorithms implement at least one unsupervised learning algorithm.

### Claim 17

A computer program product having encoded instructions embodied therein, when executed by a processor perform the steps to produce a haptic experience from an input audio signal, the steps comprising:
receiving the audio input signal simultaneously in a digital signal processing chain and an analytics processor;
processing the input audio signal in the digital signal processing chain having at least one dynamic processor;
processing the input audio signal in the analytics processor by analyzing the input audio signal characteristics to select at least one predictive algorithm and a dataset for training the predictive algorithm; and
utilizing the selected predictive algorithm to calculate at least a control parameter to control the at least one dynamic processor in the digital signal processing chain.

### Claim 18

A computer program product having encoded instructions embodied therein, when executed by a processor perform the steps to produce a haptic experience from an input audio signal, the steps comprising:
receiving the audio input signal simultaneously in a digital signal processing chain and an analytics processor;
processing the input audio signal in the digital signal processing chain having at least one dynamic processor;
processing the input audio signal in the analytics processor by analyzing the input audio signal characteristics to select at least one predictive algorithm and a dataset for training the predictive algorithm;
utilizing the selected predictive algorithm to determine the input audio signal characteristics to predict haptic data; and
producing the haptic output using one or more actuators.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein the at least one predictive algorithm is trained using the real-time data.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein the real-time data comprises of RMS values and peak values and a sequence of steps to determine a gain and/or a threshold.

### Claim 21 (depends on 18)

The computer program product of claim 18, wherein the at least one predictive algorithm is trained using both the pre-stored data and the real-time data.

### Claim 22 (depends on 18)

The computer program product of claim 18, wherein the at least one predictive algorithm implements at least one supervised learning algorithm.

### Claim 23 (depends on 18)

The computer program product of claim 18, wherein the at least one predictive algorithm implements at least one unsupervised learning algorithm.

### Claim 24 (depends on 18)

The computer program product of claim 18, wherein the at least one parameter is the gain.

### Claim 25 (depends on 18)

The computer program product of claim 18, wherein the at least one parameter is the threshold.
