# US 2017/0256269 A1 · Monaural intrusive speech intelligibility predictor unit, a hearing aid and a binaural hearing aid system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0256269 A1
- Document ID: US20170256269A1
- Status: Published application
- Kind: A1
- Application number: 15/445,472
- Filed: 2017-02-28
- Published: 2017-09-07
- Earliest claimed: 2016-03-01
- Assignees: Oticon A/S
- Applicants: Oticon A/S
- Inventors: Jesper JENSEN; Jan Mark DE HAAN; Asger Heidemann ANDERSEN
- Web page: https://patentgrep.com/patent/US20170256269A1

## CPC

- G10L 21/0364: for improving intelligibility
- A61N 1/378: Electrical supply
- G10L 21/0232: Processing in the frequency domain
- G10L 21/038: using band spreading techniques
- G10L 25/06: the extracted parameters being correlation coefficients
- G10L 25/60: for measuring the quality of voice signals
- H04R 25/505: {using digital signal processing}
- H04R 25/552: {Binaural}
- H04R 25/554: {using a wireless connection, e.g. between microphone and amplifier or using Tcoils}
- H04R 25/70: {Adaptation of deaf aid to hearing loss, e.g. initial electronic fitting}
- A61N 1/36038: {Cochlear stimulation}
- H04R 25/558: {Remote control, e.g. of amplification, frequency}
- H04R 25/606: {acting directly on the eardrum, the ossicles or the skull, e.g. mastoid, tooth, maxillary or mandibular bone, or mechanically stimulating the cochlea, e.g. at the oval window}
- H04R 2225/025: In the ear hearing aids [ITE] hearing aids
- H04R 2225/41: Detection or adaptation of hearing aid parameters or programs to listening situation, e.g. pub, forest
- H04R 2225/43: Signal processing in hearing aids to enhance the speech intelligibility
- H04R 2225/55: Communication between hearing aids and external devices via a network for data exchange
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- A61N: ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY (measurement of bioelectric currents A61B; surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body A61B18/00; anaesthetic apparatus in general A61M; incandescent lamps H01K; infrared radiators for heating H05B)
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

A monaural intrusive speech intelligibility predictor unit comprises: first and second input units for providing time-frequency representations s(k,m) and x(k,m) of noise-free and noisy and/or processed versions of a target signal, respectively, k being a frequency bin index, k=1, 2, . . . , K, and m being a time index; first and second envelope extraction units for providing time-frequency sub-band representations of the signals sj(m) and xj(m), j being a frequency sub-band index, j=1, 2, . . . , J; first and second time-frequency segment division units for dividing the time-frequency sub-band representations sj(m) and xj(m) into time-frequency segments Sm and Xm corresponding to a number N of successive samples of the sub-band signals; an intermediate speech intelligibility calculation unit adapted for providing intermediate speech intelligibility coefficients dm estimating an intelligibility of said time-frequency segment Xm, based on said time-frequency segments Sm and Xm or normalized and/or transformed versions {tilde over (S)}m, and {tilde over (X)}m thereof; and a final monaural speech intelligibility calculation unit for calculating a final monaural speech intelligibility predictor d estimating an intelligibility of said noisy and/or processed version x of the target signal by combining said intermediate speech intelligibility coefficients dm, or a transformed version thereof, over time. A hearing aid comprises a monaural, intrusive intelligibility predictor unit, and a configurable signal processor adapted to control or influence the processing of one or more electric input signals representing environment sound to maximize the final speech intelligibility predictor d. A binaural hearing aid system comprises first and second hearing aids.

## Claims

### Claim 1

A monaural speech intelligibility predictor unit adapted for receiving a target signal comprising speech in an essentially noise-free version s and in a noisy and/or processed version x, the monaural speech intelligibility predictor unit being configured to provide as an output a final monaural speech intelligibility predictor value d indicative of a listener's perception of said noisy and/or processed version x of the target signal, the monaural speech intelligibility predictor unit comprising
A first input unit for providing a time-frequency representation s(k,m) of said noise-free version s of the target signal, k being a frequency bin index, k=1, 2, . . . , K, and m being a time index;
A second input unit for providing a time-frequency representation x(k,m) of said noisy and/or processed version x of the target signal, k being a frequency bin index, k=1, 2, . . . , K, and m being a time index;
A first envelope extraction unit for providing a time-frequency sub-band representation sj(m) of the noise-free version s of the target signal representing temporal envelopes, or functions thereof, of frequency sub-band signals sj(m) of said noise-free target signal, j being a frequency sub-band index, j=1, 2, . . . , J, and m being the time index;
A second envelope extraction unit for providing a time-frequency sub-band representation xj(m) of the noisy and/or processed version x of the target signal representing temporal envelopes, or functions thereof, of frequency sub-band signals xj(m) of said noisy and/or processed version of the target signal, j=1, 2, . . . , J, and m being the time index;
A first time-frequency segment division unit for dividing said time-frequency sub-band representation sj(m) of the noise-free version s of the target signal into time-frequency segments Sm corresponding to a number N of successive samples of said sub-band signals;
A second time-frequency segment division unit for dividing said time-frequency sub-band representation xj(m) of the noisy and/or processed version x of the target signal into time-frequency segments Xm corresponding to a number N of successive samples of said sub-band signals;
A normalization and transformation unit configured to provide at least one normalization and/or transformation operation of rows and at least one normalization and/or transformation operation of columns of the time-frequency segments Sm and Xm;
An intermediate speech intelligibility calculation unit adapted for providing intermediate speech intelligibility coefficients dm estimating an intelligibility of said time-frequency segment Xm, said intermediate speech intelligibility coefficients dm being based on said essentially noise-free, normalized and/or transformed time frequency segments {tilde over (S)}m, and said noisy and/or processed, normalized and/or transformed time-frequency segments {tilde over (X)}m;
A final monaural speech intelligibility calculation unit for calculating a final monaural speech intelligibility predictor d estimating an intelligibility of said noisy and/or processed version x of the target signal by combining, e.g. by averaging, or by applying a MIN or MAX-function to, said intermediate speech intelligibility coefficients dm, or a transformed version thereof, over time.

### Claim 2 (depends on 1)

A monaural speech intelligibility predictor unit according to claim 1 comprising a voice activity detector unit for indicating whether or not or to what extent a given time-segment of the essentially noise-free version s and the noisy and/or processed version x, respectively, of the target signal comprises or is estimated to comprise speech, and providing a voice activity control signal indicative thereof.

### Claim 3 (depends on 1)

A monaural speech intelligibility predictor unit according to claim 1 comprising a voice activity detector unit for identifying time-segments of the essentially noise-free version s and the noisy and/or processed version x, respectively, of the target signal comprising or estimated to comprise speech, and wherein the monaural speech intelligibility predictor unit is configured to provide modified versions of the essentially noise-free version s and of the noisy and/or processed version x, respectively, of the target signal, said modified versions comprising only such time segments comprising speech or being estimated to comprise speech.

### Claim 4 (depends on 1)

A monaural speech intelligibility predictor unit according to claim 1 comprising a hearing loss model unit configured to apply a modification of the said noisy and/or processed version x of the target signal reflecting a deviation from normal hearing of a relevant ear of the listener to provide a modified noisy and/or processed version x of the target signal for use together with said essentially noise-free version s of the target signal as a basis for calculating the final monaural speech intelligibility predictor d.

### Claim 5 (depends on 1)

A monaural speech intelligibility predictor unit according to claim 1 wherein said hearing loss model unit is configured to add a statistically independent noise signal, which is spectrally shaped according to an audiogram of the relevant ear of the listener, to said noisy and/or processed version x of the target signal.

### Claim 6 (depends on 1)

A monaural speech intelligibility predictor unit according to claim 1 adapted to extract said temporal envelope signals xj(m) and sj(m), respectively, as
z
j

(
m
)
=
f

(
∑
k
=
k


1

(
j
)
k


2

(
j
)



z

(
k
,
m
)

2
)
,
where z represents x or s, j=1, . . . , J and m=1, . . . , M, k1(j) and k2(j) denote DFT bin indices corresponding to lower and higher cut-off frequencies of the jth sub-band, J is the number of sub-bands, and M is the number of signal frames in the signal in question, and ƒ(•) is a function.

### Claim 7 (depends on 6)

A monaural speech intelligibility predictor unit according to claim 6 wherein the function f(•)-f(w), where w represents
(
∑
k
=
k


1

(
j
)
k


2

(
j
)



z

(
k
,
m
)

2
)
,
is selected among the following functions
ƒ(w)=w representing the identity
ƒ(w)=w2 providing power envelopes,
ƒ(w)=2·log w or ƒ(w)=wβ, 0<β<2, allowing the modelling of the compressive non-linearity of the healthy cochlea,
or combinations thereof.

### Claim 8 (depends on 1)

A monaural speech intelligibility predictor unit according to claim 1 wherein said first and second time-frequency segment division units are configured to divide said time-frequency representations sj(m) and xj(m), respectively, into segments in the form of spectrograms corresponding to N successive samples of all sub-band signals, wherein the mth segment is defined by the J×N matrix
Z
m
=
[
z
1

(
m
-
N
+
1
)
…
z
1

(
m
)
⋮
⋮
z
J

(
m
-
N
+
1
)
…
z
J

(
m
)
]
.
where z, Z represents s, S or x, X, respectively.

### Claim 9 (depends on 1)

A monaural speech intelligibility predictor unit according to claim 1 comprising
A first normalization and/or transformation unit adapted for providing normalized and/or transformed versions {tilde over (S)}m of said time-frequency segments Sm;
A second normalization and/or transformation unit adapted for providing normalized and/or transformed versions {tilde over (X)}m of said time-frequency segments Xm;

### Claim 10 (depends on 9)

A monaural speech intelligibility predictor unit according to claim 9, wherein first and second normalization and/or transformation units are configured to apply one or more of the following algorithms to the time-frequency segments Xm and Sm, respectively, commonly denoted Zm, where sub-script, time index m is skipped for simplicity in the following expressions:
Normalization of rows to zero mean:
g1(Z)=Z−μzr1T,
where μzr is a J×1 vector whose j′th entry is the mean of the j′th row of Z, hence the superscript r in μzr, where 1 denotes an N×1 vector of ones, and where superscript T denotes matrix transposition;
Normalization of rows to unit-norm:
g2(Z)=Dr(Z)Z,
where Dr(Z)=diag(└1/√{square root over (Z(1,:)Z(1,:)H)} . . . 1/√{square root over (Z(J,:)Z(J,:)H)}┘), where diag(•) is a diagonal matrix with the elements of the arguments on the main diagonal, and where Z(j,:) denotes the j′th row of Z, such that D′(Z) is a J×J diagonal matrix with the inverse norm of each row on the main diagonal, and zeros elsewhere, the superscript H denotes Hermitian transposition, and pre-multiplication with D′(Z) normalizes the rows of the resulting matrix to unit-norm;
Fourier transformation applied to each row
g3(Z)=ZF,
where F is an N×N Fourier matrix;
Fourier transformation applied to each row followed by computing the magnitude of the resulting complex-valued elements
g4=|ZF|
where |•| computes the element-wise magnitudes;
The identity operator
g5(Z)=Z.
Normalization of columns to zero mean:
h1(Z)=Z−1μzcT,
where μzc is a N×1 vector whose ith entry is the mean of the ith row of Z, and where 1 denotes a J×1 vector of ones;
Normalization of columns to unit-norm:
h2(Z)=ZDc(Z),
where Dc(Z)=diag(└1/√{square root over (Z(:,1)HZ(:,1))} . . . 1/√{square root over (Z(:,N)HZ(:,N))}┘, where Z(:,n) denotes the n′th row of Z, such that Dc(z) is a diagonal N×N matrix with the inverse norm of each column on the main diagonal, and zeros elsewhere, and where a post-multiplication with Dc(Z) normalizes the rows of the resulting matrix to unit-norm.

### Claim 11 (depends on 1)

A monaural speech intelligibility predictor unit according to claim 1 wherein the intermediate speech intelligibility calculation unit is adapted to determine said intermediate speech intelligibility coefficients dm in dependence on a, e.g. linear, sample correlation coefficient d(a,b) of the elements in two K×1 vectors a and b, d(a,b) being defined by:
d

(
a
,
b
)
=
∑
k
=
1
K


(
a

(
k
)
-
μ
a
)

(
b

(
k
)
-
μ
b
)
∑
k
=
1
K


(
a

(
k
)
-
μ
a
)
2

(
b

(
k
)
-
μ
b
)
2
,

where
μ
a
=
1
K

∑
k
=
1
K


a

(
k
)


and


μ
b
=
1
K

∑
k
=
1
K

b

(
k
)
,
where k is the index of the vector entry and K is the vector dimension.

### Claim 12 (depends on 11)

A monaural speech intelligibility predictor unit according to claim 11 wherein the intermediate intelligibility index dm is defined as
the average sample correlation coefficient of all columns in Sm and Xm, or {tilde over (S)}m and {tilde over (X)}m, respectively, i.e.,
d
m
=
1
N

∑
n
=
1
N


d

(
S
~
m

(
:
,
n
)
,
X
~
m

(
:
,
n
)
)
,
n being a column index, or as
the average sample correlation coefficient of all rows in Sm and Xm, or {tilde over (S)}m and {tilde over (X)}m, i.e.,
d
m
=
1
J

∑
j
=
1
J


d

(
S
~
m

(
j
,
:
)
T
,
X
~
m

(
j
,
:
)
T
)
,
j being a row index, or as
the sample correlation coefficient of all elements in Sm and Xm, or {tilde over (S)}m and {tilde over (X)}m, i.e.,
dm=d({tilde over (S)}m(:),{tilde over (X)}m(:)),
where the notation Sm(:) and Xm(:), or {tilde over (S)}m(:) and {tilde over (X)}m(:), represents NJ×1 vectors formed by stacking the columns of the respective matrices.

### Claim 13 (depends on 1)

A monaural speech intelligibility predictor unit according to claim 1 wherein the final speech intelligibility calculation unit is adapted to calculate the final speech intelligibility predictor d from the intermediate speech intelligibility coefficients dm, optionally transformed by a function u(dm), as an average over time of said information signal x:
d
=
1
M

∑
m
=
1
M


u

(
d
m
)
.
where M represents the duration in time units of the speech active parts of said noisy and/or processed version x of the target signal.

### Claim 14 (depends on 13)

A monaural speech intelligibility predictor unit according to claim 13 wherein function u(dm) is defined as
u

(
d
m
)
=
log

(
1
1
-
d
m
2
)
,
or as
u(dm)=dm.

### Claim 15 (depends on 1)

A hearing aid adapted for being located at or in left or right ears of a user, or for being fully or partially implanted in the head of the user, the hearing aid comprising a monaural speech intelligibility predictor unit according to claim 1.

### Claim 16 (depends on 15)

A hearing aid according to claim 15 configured to adaptively modify the processing of an input signal to the hearing aid to maximize the monaural speech intelligibility predictor d, to enhance the user's intelligibility of an output signal of the hearing aid presented to the user.

### Claim 17 (depends on 15)

A binaural hearing system comprising left and right hearing aids according to claim 15, wherein each of the left and right hearing aids comprises antenna and transceiver circuitry for allowing a communication link to be established and information to be exchanged between said left and right hearing aids, the binaural hearing system further comprising a binaural speech intelligibility prediction unit for providing a final binaural speech intelligibility measure dbinaural of the predicted speech intelligibility of the user, when exposed to said sound input, based on the monaural speech intelligibility predictor values dleft, dright of the respective left and right hearing aids.

### Claim 18 (depends on 17)

A binaural hearing system according to claim 17, wherein the respective configurable signal processors of the left and right hearing aids are adapted to control or influence the processing of the respective electric input signals to maximize said final binaural speech intelligibility measure dbinaural.

### Claim 19

A method of providing a monaural speech intelligibility predictor for estimating a user's ability to understand an information signal x comprising a noisy and/or processed version of a target speech signal, the method comprising
providing a time-frequency representation s(k,m) of said noise-free version s of the target signal, k being a frequency bin index, k=1, 2, . . . , K, and m being a time index;
providing a time-frequency representation x(k,m) of said noisy and/or processed version x of the target signal, k being a frequency bin index, k=1, 2, . . . , K, and m being a time index;
providing a time-frequency sub-band representation sj(m) of the noise-free version s of the target signal representing temporal envelopes, or functions thereof, of frequency sub-band signals sj(m) of said noise-free target signal, j being a frequency sub-band index, j=1, 2, . . . , J, and m being the time index;
providing a time-frequency sub-band representation xj(m) of the noisy and/or processed version x of the target signal representing temporal envelopes, or functions thereof, of frequency sub-band signals xj(m) of said noisy and/or processed version of the target signal, j=1, 2, . . . , J, and m being the time index;
dividing said time-frequency sub-band representation sj(m) of the noise-free version s of the target signal into time-frequency segments Sm corresponding to a number N of successive samples of said sub-band signals;
dividing said time-frequency sub-band representation xj(m) of the noisy and/or processed version x of the target signal into time-frequency segments Xm corresponding to a number N of successive samples of said sub-band signals;
providing at least one normalization and/or transformation operation of rows and at least one normalization and/or transformation operation of columns of the time-frequency segments Sm and Xm;
providing intermediate speech intelligibility coefficients dm estimating an intelligibility of said time-frequency segment Xm, said intermediate speech intelligibility coefficients dm being based on said essentially noise-free, normalized and/or transformed time frequency segments {tilde over (S)}m, and said noisy and/or processed, normalized and/or transformed time-frequency segments {tilde over (X)}m;
calculating a final monaural speech intelligibility predictor d estimating an intelligibility of said noisy and/or processed version x of the target signal by combining, e.g. by averaging or applying a MIN or MAX-function, said intermediate speech intelligibility coefficients dm, or a transformed version thereof, over time.

### Claim 20 (depends on 19)

A non-transitory computer-readable medium storing a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of claim 19.
