TOWARDS BALANCED SPECTRAL RECONSTRUCTION: SPECTRALLY ADAPTIVE LOSS FOR STREAMING SPEECH ENHANCEMENT
Haixin Zhao and Nilesh Madhu | IDLab, Ghent University - imec | Submitted to IWAENC 2026
Correction
The operations in the spectrally adaptive loss formulation should be applied in the following order: min-max normalisation followed by sigmoid mapping. The corrected formulation is
ℒAdp = 0.7·ℒMag + λadp·Fs(σ(N(Et[log|S|])))·ℒPha.
Live Demo: Real-Time Speech Enhancement - Right in Your Browser
Our model runs fully in real time on plain CPU - no GPU, no NPU, no special hardware. It runs smoothly even on a $35 (+ $1 recommended cooling system - see image below) edge device like a Raspberry Pi 4B, and works directly in the browser on your phone, laptop, or desktop. Click the link below and hear it enhance your own voice live, on whatever device you have in your hand right now.
https://aspire.ugent.be/demos/real_time_SE/dist/ Click it!
Audio Samples
This page provides audio samples and spectrogram visualisations accompanying our submission. We compare three loss functions, all trained on HyST-Net for streaming speech enhancement:
- ℒMR-Mix: baseline mixed magnitude + phase-aware loss
- ℒMR-Sig: proposed sigmoid-weighted loss applying a fixed frequency-dependent modulation to the phase-aware contribution
- ℒMR-Adp: proposed spectrally adaptive loss applying a signal-dependent modulation derived from the ground-truth log-magnitude spectrogram
Across the samples, both proposed losses recover high-frequency energy attenuated by the baseline to varying extents, as visible in the upper spectrogram areas and perceptible as improved brightness. For example, in Sample 1, the word "hundred" at the onset sounds noticeably clearer and brighter under ℒMR-Adp compared to both ℒMR-Mix and ℒMR-Sig, reflecting its further improvement in mid-frequency reconstruction through signal-dependent weighting.
| Samples | Noisy | ℒMR-Mix | ℒMR-Sig | ℒMR-Adp | Clean |
|---|---|---|---|---|---|
| Sample 1 |
|
|
|
|
|
| Sample 2 |
|
|
|
|
|
| Sample 3 |
|
|
|
|
|
| Sample 4 |
|
|
|
|
|
| Sample 5 |
|
|
|
|
|
| Sample 6 |
|
|
|
|
|
Limitation Cases
The following samples illustrate a limitation observed in a minority of cases. While both ℒMR-Sig and ℒMR-Adp recover mid-to-high frequency magnitude energy, certain noise types can occasionally introduce a subtle hissing artefact in the high-frequency background in the high-frequency background. This points to a direction for future work.
| Samples | Noisy | ℒMR-Mix | ℒMR-Sig | ℒMR-Adp | Clean |
|---|---|---|---|---|---|
| Sample f1 |
|
|
|
|
|
| Sample f2 |
|
|
|
|
|