Spectral masking analysis · Django + NumPy

Where does the music bury the voice?

SomaSonic answers that per frequency band, per time window — from the raw samples, not from a listening session. Upload a narration stem and the bed that plays underneath it, and it returns the map of exactly where the two are fighting.

CLEAR −40 dBHEADROOM −6 dBSEVERE CLASH

The tool, running

2 min 22 s · captions included

Download the screencast MP4 · 1.1 MB · 1280×720

The interface

Four states, straight from the recording

SomaSonic dashboard on a dark teal ground: a welcome panel with a New Stem button, and empty New pairing, Your pairings and Your stems cards.
The workspace. Stems, pairings and past analyses in one place. The empty state says what is missing and links straight to the fix rather than showing a blank panel.
Upload Production Stem form with a title field, a role selector set to Music Bed slash Ambient Background Layer, a locked file showing music_bed.wav at 1.01 megabytes, and a progress bar reading Downsampling pass successful.
Upload, with the work done before the upload. The browser decodes the file, downmixes it to mono, resamples to 16 kHz and converts to Int16 PCM — so what crosses the network is 5.5 MB instead of 33 MB. The progress bar is reporting that local pass, not the transfer.
Spectral Clash Analysis Room for Episode 4 showing a clash severity of 100 percent, total clash time 7.07 seconds, and a frequency masking timeline map: a dark canvas with vertical teal and magenta streaks, low bands at the bottom and high bands at the top.
The result. Time runs left to right, frequency bottom to top. Cool means the voice is clear; hot means the bed is within 6 dB of it. The severity figure is the share of speaking frames that are contested — silence between words cannot be masked, so it is never counted.
The same analysis screen at a narrow width, with the heading, legend and heatmap reflowing into a single column.
The same screen, narrow. The heatmap keeps the worst value in each column as it reduces, so the picture stays honest at any width — it never smooths a clash away to fit.

How it works

Client decimation → NumPy STFT → canvas

  1. Decode in the browser, not on the serverAn OfflineAudioContext downmixes to mono and resamples to 16 kHz Int16 PCM before anything is sent. That is what keeps the backend down to two dependencies — no ffmpeg, no librosa, no codec binaries in the deploy.
  2. Trust the samples, not the metadataThe server reads the byte array with np.frombuffer(..., '<i2') and derives duration by counting samples, so a file lying in its header cannot shift the timeline.
  3. Short-time Fourier transformA Hanning window, N=1024, hop 512 — 31.25 frames per second. A three-minute track becomes 5,624 analysable frames.
  4. Two gates, because a naive score is meaninglessFrames below the energy floor are skipped — silence cannot be masked, and without this gate the quiet gaps dominate the score. Then anything more than 40 dB under a frame's own peak is discarded, so spectral leakage is not read as a clash.
  5. Eight log-spaced bands0–8016 Hz with widening bins (20/20/25/35/60/90/140/123) to match how hearing actually resolves pitch, scored against a named HEADROOM_DB = −6.0 threshold.
  6. Render without melting the browser44,992 stored values reduce to 6,240 canvas cells — one column per pixel, worst value wins — drawn with fillRect in about 25 ms. As DOM elements it would have frozen the render tree.

Why this problem

It came from a real failure

I run an autonomous pipeline that renders long-form documentary films — generated narration over a generated music bed. Its worst recurring defect was never a crash: it was a finished film where the bed sat on top of the voice, and nothing in the system could see it, because loudness targets are met by a mix that is still unintelligible. A single number for the whole track cannot find a clash that lives in one band for seven seconds.

SomaSonic is that missing measurement built as its own tool: not “is this loud enough”, but where, and for how long, is the bed inside the voice's headroom. The severity figure in the screenshot reads 100% on a deliberately bad pairing — the clearest way to show a detector actually detecting.

Academic honesty. SomaSonic is my capstone for HarvardX CS50W (Web Programming with Python and JavaScript), submitted in September 2026 and currently in grading — not a completed credential.

The source repository stays private under CS50's policy on publishing coursework. This page shows the interface and the design decisions; it does not distribute the code. Happy to walk through the implementation live.