Spectral masking analysis · Django + NumPy

Where does the music bury the voice?

SomaSonic answers that per frequency band, per time window — from the raw samples, not from a listening session. Upload a narration stem and the bed that plays underneath it, and it returns the map of exactly where the two are fighting.

CLEAR −40 dBHEADROOM −6 dBSEVERE CLASH

The tool, running

2 min 22 s · captions included

Download the screencast MP4 · 1.1 MB · 1280×720

The interface

Four states, straight from the recording

SomaSonic dashboard: a New pairing panel with narration and music-bed selectors, a Your pairings row showing Ngorongoro Ep2 at 48 percent severity beside a small colour ramp, and the uploaded stems listed below.
The workspace. Every pairing keeps its score, and the score carries the same colour ramp the analysis screen uses — so a number and its colour can never disagree between two views.
Upload Production Stem form with a title field, a role selector set to Music Bed slash Ambient Background Layer, a locked file showing music_bed.wav at 1.01 megabytes, and a progress bar reading Downsampling pass successful.
Upload, with the work done before the upload. The browser decodes the file, downmixes to mono, resamples to 16 kHz and converts to Int16 PCM — so what crosses the network is 5.5 MB instead of 33 MB. The progress bar is reporting that local pass, not the transfer.
Spectral clash analysis for Ngorongoro Ep2 opening: clash severity 48 percent with a marker on a colour ramp, 27.42 seconds of speech contested, and a frequency masking timeline with a labelled frequency axis from 312 hertz to 8 kilohertz and a time axis from 0 to 60 seconds. The magenta clash concentrates in the lowest band.
The result. Time runs left to right (0–60 s), frequency bottom to top (312 Hz–8 kHz). Cool means the voice is clear; hot means the bed is within 6 dB of it. Read the bottom row: this bed is fighting the narration almost entirely below 312 Hz — which is a fix you can act on, and exactly what a single loudness number cannot tell you. The vertical breaks are the gaps between spoken lines; silence is never scored, because silence cannot be masked.
The analysis screen on a 390 pixel phone viewport: the header, severity meter, legend and heat map stack into one column, the frequency axis stays beside the map, and the time axis drops to fewer labels.
The same screen on a phone — a real 390 px viewport, not a scaled-down desktop. The map keeps the worst value in each column as it reduces rather than averaging, so a tenth-of-a-second clash survives the squeeze; the time axis just thins its labels to fit.

How it works

Client decimation → NumPy STFT → canvas

  1. Decode in the browser, not on the serverAn OfflineAudioContext downmixes to mono and resamples to 16 kHz Int16 PCM before anything is sent. That is what keeps the backend down to two dependencies — no ffmpeg, no librosa, no codec binaries in the deploy.
  2. Trust the samples, not the metadataThe server reads the byte array with np.frombuffer(..., '<i2') and derives duration by counting samples, so a file lying in its header cannot shift the timeline.
  3. Short-time Fourier transformA Hanning window, N=1024, hop 512 — 31.25 frames per second. A three-minute track becomes 5,624 analysable frames.
  4. Two gates, because a naive score is meaninglessFrames below the energy floor are skipped — silence cannot be masked, and without this gate the quiet gaps dominate the score. Then anything more than 40 dB under a frame's own peak is discarded, so spectral leakage is not read as a clash.
  5. Eight log-spaced bands0–8016 Hz with widening bins (20/20/25/35/60/90/140/123) to match how hearing actually resolves pitch, scored against a named HEADROOM_DB = −6.0 threshold.
  6. Render without melting the browser44,992 stored values reduce to 6,240 canvas cells — one column per pixel, worst value wins — drawn with fillRect in about 25 ms. As DOM elements it would have frozen the render tree.

Why this problem

It came from a real failure

I run an autonomous pipeline that renders long-form documentary films — generated narration over a generated music bed. Its worst recurring defect was never a crash: it was a finished film where the bed sat on top of the voice, and nothing in the system could see it, because loudness targets are met by a mix that is still unintelligible. A single number for the whole track cannot find a clash that lives in one band for seven seconds.

SomaSonic is that missing measurement built as its own tool: not “is this loud enough”, but where, and for how long, is the bed inside the voice's headroom. The run shown above is real material from that pipeline — a Ngorongoro episode's narration against its own music bed. It scores 48%, with 27.4 seconds of speech contested, almost all of it under 312 Hz.

Academic honesty. SomaSonic is my capstone for HarvardX CS50W (Web Programming with Python and JavaScript), submitted in September 2026 and currently in grading — not a completed credential.

The source repository stays private under CS50's policy on publishing coursework. This page shows the interface and the design decisions; it does not distribute the code. Happy to walk through the implementation live.