ddidderr/threejs-sound-sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add spatial audio simulator with camera-as-listener model

Browse Source 
Implements the simulator described in README.md: a 3D room in which a
loudspeaker can be moved and aimed, heard binaurally from the camera's
position, with every acoustic term shown live so the user can read why it
sounds the way it does.

Only README.md was previously committed. An earlier AI-generated
implementation existed in the working tree but was never committed; it is
superseded rather than modified. That implementation did not work:
renderer.setSize() was never called and the stylesheet had no rule for
html, body or the canvas container, so the WebGL view rendered into a
300x150 box in the corner of an unstyled page. Its acoustics were also
wrong in ways that voided the stated goal, which is what motivated a
rewrite rather than a repair:

  - AudioContext.listener orientation (forwardX/Y/Z, upX/Y/Z) was never
    written, only position. Orbiting the camera therefore produced no
    change in the sound field at all.
  - The listener was a separate avatar mesh, not the camera, contradicting
    the README. Its syncFromCamera() had zero callers.
  - The reverb send was tapped after the distance, cone and occlusion
    gains, making the direct-to-reverberant ratio mathematically constant.
    Walking away sounded like a fader move rather than like distance.
  - Doppler was derived from a single-frame velocity finite difference
    smoothed with a per-frame (not per-second) lerp, so any drag produced
    an octave-wide pitch chirp and the response varied with refresh rate.

Design decisions worth recording, since each has a cheaper alternative
that was rejected:

The camera IS the Web Audio listener. Position and orientation are both
published every frame. OrbitControls panning is disabled on purpose: it
moves the orbit target without moving the camera, which would silently
desync the ear from the view and break that invariant.

Doppler is emergent, not computed. A DelayNode holds distance /
speedOfSound; chasing it resamples the signal exactly as air does. This
removes velocity estimation from the codebase entirely, which is what
eliminated the drag-chirp class of bug. The reported ratio is 1 - dD/dt,
the shift a delay line actually produces. The textbook moving-source form
1/(1 - v/c) agrees only to first order and has a pole that inverts the
sign of the shift when a source closes fast.

The reverb send is tapped BEFORE distance, directivity and occlusion,
because a room's reverberant field is roughly uniform: it depends on how
much power the source radiates, not on where the listener stands. This is
what makes the critical distance audible, and it is the one thing the
simulator teaches that a fader cannot fake.

Room acoustics are derived from geometry via Sabine (T60 = 0.161V/Sa)
rather than from named presets, so the room-size sliders genuinely change
the sound. The impulse response is normalised to unit energy per channel,
not unit peak, because a convolution's output level follows total energy;
peak normalisation would make a long tail far louder than a short one and
the send gain would stop meaning anything.

Source presets are built additively from a bounded harmonic series instead
of being sampled from ideal waveforms, leaving an octave of clean headroom
so Doppler can pitch them up without folding harmonics back down as
aliasing.

User-supplied audio files are summed to mono. A stereo mix fed into the
panner would leave part of the image fixed to the listener's ears no
matter where the source moved, which is precisely the illusion this app
exists to break.

Dark theme only. The viewport is a lit 3D room; a light chrome around it
reads as a white frame on a dark photograph, and re-lighting the scene for
a light theme would need a second set of materials for no real gain.

User-visible behaviour, relative to the uncommitted prior version:

  - The 3D view fills its pane and is sized on construction plus via a
    ResizeObserver.
  - Orbiting the view swings the sound between the ears (measured at
    roughly 14 dB of L-R swing across a full orbit in an anechoic room).
  - Walking away attenuates the direct path while the reverberant field
    holds steady, and a HUD chip reports which of the two dominates.
  - A signal-path table lists every stage between source and ear in dB on
    one shared -60..0 dB scale. The stage losses sum exactly to the total;
    air absorption deliberately reports no dB because it has no broadband
    loss, only a cutoff.
  - Audio files can be loaded by drop or picker and behave as the source.
  - Three guided demos, a first-run explainer, and a keyboard model.
  - Leaked agent artifacts are gone: the panel header no longer reads
    "M4 OVERLAY", sections are no longer lettered "A.", "B.", and the
    milestone scratch directories and docs are deleted.

Known limitations, also listed in README.md:

  - Reverb is a single static impulse response per room; it does not vary
    with listener position, so walking into a corner is not audible.
  - No precedence effect. Real ears localise well beyond the critical
    distance because the first wavefront wins; here localisation degrades
    with the direct-to-reverberant ratio, so a very live room smears the
    image more than it would in life.
  - Occlusion tests one obstacle and models diffraction as a fixed
    broadband loss rather than as a function of the obstacle's size.
  - HRTF quality is whatever the browser's PannerNode provides.
  - Loaded files loop with an audible seam; the crossfade used on the
    generated presets would be wrong for music.

Test Plan

  npm test          116 unit tests. Physics and room acoustics are pure
                    functions. The audio graph is tested against a mock
                    AudioContext that records connections, so the suite
                    can assert topology directly: that the reverb send
                    hangs off the delay rather than the direct gain, and
                    that listener forward/up are published and not just
                    position. Both were silently wrong before.
  npm run test:e2e  47 checks driving real Chrome; starts its own dev
                    server. Covers what a mock cannot: that the view is
                    actually sized, that orbiting changes the ear balance,
                    that walking away attenuates the direct path while the
                    room holds steady, that each demo does what its
                    caption claims, that the panel follows the demos
                    rather than silently reverting them, and that a real
                    WAV decoded by the browser spatialises like any other
                    source while a non-audio file fails visibly.
  npm run build     tsc plus a production bundle, split so three.js gets
                    its own long-lived chunk (470 kB three, 99 kB app).

Manual check: load the page in a browser with headphones, press
"Enable audio & start", then press 3 and walk away with W/S. The Direct
row should fall while Room (wet) stays flat, and the HUD chip should flip
to room-dominant as you cross the ring drawn on the floor.

Refs: ORIGINAL_REQUEST.md
Docs: README.md
This commit is contained in:
ddidderr
2026-07-26 14:50:23 +02:00
parent acf442ab41
commit 1a827b8b0d
46 changed files with 9468 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/engine.test.ts
+440
View File
@@ -0,0 +1,440 @@
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { AudioEngine, DEFAULT_SETTINGS, SpatialInput } from '../src/audio/AudioEngine';
import { MockAudioContext, MockParam, installMockAudio } from './mockAudioContext';
let audio: ReturnType<typeof installMockAudio>;
beforeEach(() => {
audio = installMockAudio();
});
afterEach(() => {
audio.restore();
});
const ctxOf = () => audio.instances[0] as MockAudioContext;
function poseAt(x: number, z = 0, extra: Partial<SpatialInput> = {}): SpatialInput {
return {
sourcePos: { x: 0, y: 1.5, z: 0 },
sourceForward: { x: 0, y: 0, z: -1 },
listenerPos: { x, y: 1.5, z },
listenerForward: { x: 0, y: 0, z: -1 },
listenerUp: { x: 0, y: 1, z: 0 },
occlusion: 0,
...extra,
};
}
/** Settles the delay-line smoothing so telemetry reflects a steady state. */
function settle(engine: AudioEngine, pose: SpatialInput, frames = 400) {
let telemetry = engine.updateSpatial(pose, 1 / 60);
for (let i = 0; i < frames; i++) telemetry = engine.updateSpatial(pose, 1 / 60);
return telemetry;
}
describe('graph topology', () => {
it('wires the documented signal path', async () => {
const engine = new AudioEngine();
await engine.start();
const edges = ctxOf().edges();
expect(edges).toContain('gain->delay'); // sourceGain -> delay
expect(edges).toContain('delay->biquad'); // -> air absorption
expect(edges).toContain('biquad->biquad'); // air -> occlusion
expect(edges).toContain('biquad->gain'); // occlusion -> directGain
expect(edges).toContain('gain->panner');
expect(edges).toContain('gain->convolver'); // reverbSend -> convolver
expect(edges).toContain('convolver->gain'); // wet -> master
expect(edges).toContain('gain->compressor'); // master -> limiter
expect(edges).toContain('compressor->analyser');
expect(edges).toContain('analyser->destination');
});
it('taps the reverb send before distance, cone and occlusion', async () => {
const engine = new AudioEngine();
await engine.start();
const ctx = ctxOf();
// The wet send must hang off the delay, not off the direct gain: a room's
// reverberant field does not get quieter because you walked away.
const feedsConvolver = ctx.connections.find((c) => c.to.kind === 'convolver');
const sendSource = ctx.connections.find((c) => c.to === feedsConvolver?.from);
expect(sendSource?.from.kind).toBe('delay');
});
it('neuters the panner so only our own physics sets level', async () => {
const engine = new AudioEngine();
await engine.start();
const panner = ctxOf().nodeOf<never>('panner') as unknown as Record<string, unknown>;
expect(panner.panningModel).toBe('HRTF');
expect(panner.rolloffFactor).toBe(0);
expect(panner.coneOuterGain).toBe(1);
});
});
describe('listener orientation', () => {
it('publishes forward and up, not just position', async () => {
const engine = new AudioEngine();
await engine.start();
const listener = ctxOf().listener;
engine.updateSpatial(
{
...poseAt(5),
listenerForward: { x: 1, y: 0, z: 0 },
listenerUp: { x: 0, y: 1, z: 0 },
},
1 / 60
);
expect(listener.forwardX.value).toBeCloseTo(1, 6);
expect(listener.forwardZ.value).toBeCloseTo(0, 6);
expect(listener.upY.value).toBeCloseTo(1, 6);
expect(listener.forwardX.events.length).toBeGreaterThan(0);
});
it('tracks a turning head', async () => {
const engine = new AudioEngine();
await engine.start();
const listener = ctxOf().listener;
engine.updateSpatial({ ...poseAt(5), listenerForward: { x: 0, y: 0, z: -1 } }, 1 / 60);
expect(listener.forwardZ.value).toBeCloseTo(-1, 6);
engine.updateSpatial({ ...poseAt(5), listenerForward: { x: 0, y: 0, z: 1 } }, 1 / 60);
expect(listener.forwardZ.value).toBeCloseTo(1, 6);
});
it('normalises whatever it is handed', async () => {
const engine = new AudioEngine();
await engine.start();
const listener = ctxOf().listener;
engine.updateSpatial({ ...poseAt(5), listenerForward: { x: 0, y: 0, z: -9 } }, 1 / 60);
expect(listener.forwardZ.value).toBeCloseTo(-1, 6);
});
it('keeps the panner on the source', async () => {
const engine = new AudioEngine();
await engine.start();
const panner = ctxOf().nodeOf('panner') as unknown as Record<string, MockParam>;
engine.updateSpatial(
{ ...poseAt(5), sourcePos: { x: -3, y: 2, z: 7 } },
1 / 60
);
expect(panner.positionX.value).toBeCloseTo(-3, 6);
expect(panner.positionZ.value).toBeCloseTo(7, 6);
});
});
describe('gain staging', () => {
it('drives the direct gain from distance, cone and occlusion together', async () => {
const engine = new AudioEngine();
await engine.start();
const ctx = ctxOf();
const directGain = ctx.feederOf('gain', 'panner');
const telemetry = engine.updateSpatial(poseAt(4), 1 / 60);
const expected = Math.pow(10, telemetry.directGainDb / 20);
expect(directGain.gain.value).toBeCloseTo(expected, 5);
});
it('keeps the reverb send level independent of listener distance', async () => {
const engine = new AudioEngine();
await engine.start();
const send = ctxOf().feederOf('gain', 'convolver');
engine.updateSpatial(poseAt(2), 1 / 60);
const near = send.gain.value;
engine.updateSpatial(poseAt(40), 1 / 60);
const far = send.gain.value;
expect(near).toBeGreaterThan(0);
expect(far).toBeCloseTo(near, 6);
});
it('applies master volume on its own stage, not folded into the physics', async () => {
const engine = new AudioEngine();
await engine.start();
const ctx = ctxOf();
const directGain = ctx.feederOf('gain', 'panner');
const master = ctx.feederOf('gain', 'compressor');
const before = (engine.updateSpatial(poseAt(4), 1 / 60), directGain.gain.value);
engine.setMasterVolume(0.25);
engine.updateSpatial(poseAt(4), 1 / 60);
expect(master.gain.value).toBeCloseTo(0.25, 6);
// Changing the volume must not move the acoustic gain, or the signal-path
// readout would stop describing the physics.
expect(directGain.gain.value).toBeCloseTo(before, 6);
});
it('lowers the direct level as the listener walks away', async () => {
const engine = new AudioEngine();
await engine.start();
const near = engine.updateSpatial(poseAt(2), 1 / 60).directGainDb;
const far = engine.updateSpatial(poseAt(20), 1 / 60).directGainDb;
expect(far).toBeLessThan(near - 15);
});
it('attenuates and muffles a blocked path', async () => {
const engine = new AudioEngine();
await engine.start();
const clear = engine.updateSpatial(poseAt(6, 0, { occlusion: 0 }), 1 / 60);
const blocked = engine.updateSpatial(poseAt(6, 0, { occlusion: 1 }), 1 / 60);
expect(blocked.directGainDb).toBeLessThan(clear.directGainDb);
expect(blocked.occlusionCutoff).toBeLessThan(clear.occlusionCutoff / 10);
});
it('attenuates when the source is aimed away', async () => {
const engine = new AudioEngine();
await engine.start();
const onAxis = engine.updateSpatial(
{ ...poseAt(0, -6), sourceForward: { x: 0, y: 0, z: -1 } },
1 / 60
);
const offAxis = engine.updateSpatial(
{ ...poseAt(0, -6), sourceForward: { x: 0, y: 0, z: 1 } },
1 / 60
);
expect(offAxis.coneGainDb).toBeLessThan(onAxis.coneGainDb - 10);
});
});
describe('propagation and doppler', () => {
it('settles the delay line on the true time of flight', async () => {
const engine = new AudioEngine();
await engine.start();
const telemetry = settle(engine, poseAt(34.3));
// 34.3 m at 343 m/s is 100 ms.
expect(telemetry.timeOfFlightMs).toBeCloseTo(100, 0);
});
it('reports no shift once the geometry is still', async () => {
const engine = new AudioEngine();
await engine.start();
const telemetry = settle(engine, poseAt(10));
expect(telemetry.dopplerRatio).toBeCloseTo(1, 3);
expect(Math.abs(telemetry.dopplerCents)).toBeLessThan(2);
});
it('raises pitch as the gap closes and lowers it as the gap opens', async () => {
const engine = new AudioEngine();
await engine.start();
settle(engine, poseAt(30));
// Sweep the listener inwards: the delay shortens, so pitch must rise.
let approaching = 1;
for (let d = 30; d > 10; d -= 0.5) {
approaching = engine.updateSpatial(poseAt(d), 1 / 60).dopplerRatio;
}
expect(approaching).toBeGreaterThan(1.0005);
let receding = 1;
for (let d = 10; d < 30; d += 0.5) {
receding = engine.updateSpatial(poseAt(d), 1 / 60).dopplerRatio;
}
expect(receding).toBeLessThan(0.9995);
});
it('stays silent about doppler when propagation is switched off', async () => {
const engine = new AudioEngine({ propagationEnabled: false });
await engine.start();
settle(engine, poseAt(30));
for (let d = 30; d > 10; d -= 0.5) engine.updateSpatial(poseAt(d), 1 / 60);
const telemetry = engine.updateSpatial(poseAt(10), 1 / 60);
expect(telemetry.timeOfFlightMs).toBeCloseTo(0, 3);
expect(telemetry.dopplerRatio).toBeCloseTo(1, 6);
});
it('reports the shift a delay line actually produces, 1 − dD/dt', async () => {
const engine = new AudioEngine();
await engine.start();
settle(engine, poseAt(30));
// Close at a steady rate and compare the reported ratio against 1 − D'
// reconstructed from the reported time of flight.
let previousFlight = engine.updateSpatial(poseAt(30), 1 / 60).timeOfFlightMs;
let checked = 0;
for (let d = 29.5; d > 12; d -= 0.5) {
const t = engine.updateSpatial(poseAt(d), 1 / 60);
const delayRate = (t.timeOfFlightMs - previousFlight) / 1000 / (1 / 60);
previousFlight = t.timeOfFlightMs;
expect(t.dopplerRatio).toBeCloseTo(1 - delayRate, 6);
checked++;
}
expect(checked).toBeGreaterThan(10);
});
it('keeps the shift on the correct side of unity even when closing hard', async () => {
// The textbook 1/(1 − v/c) form has a pole that flips sign here, reporting
// a two-octave drop for a source rushing towards the listener.
const engine = new AudioEngine({ speedOfSound: 80 });
await engine.start();
settle(engine, poseAt(40));
for (let d = 39; d > 2; d -= 3) {
const t = engine.updateSpatial(poseAt(d), 1 / 60);
expect(t.dopplerRatio).toBeGreaterThanOrEqual(1);
expect(t.closingSpeed).toBeGreaterThan(0);
}
});
it('does not saturate the delay line at the slowest speed of sound', async () => {
const engine = new AudioEngine({ speedOfSound: 80, room: { width: 60, height: 60, depth: 60 } });
await engine.start();
// Room diagonal is ~104 m; at 80 m/s that is 1.3 s of flight.
const telemetry = settle(engine, poseAt(100), 2000);
expect(telemetry.timeOfFlightMs).toBeCloseTo(1250, -1);
});
it('never lets a violent jump produce an absurd pitch', async () => {
const engine = new AudioEngine();
await engine.start();
settle(engine, poseAt(60));
// Teleport the listener: a naive velocity estimate would chirp an octave.
const telemetry = engine.updateSpatial(poseAt(0.5), 1 / 60);
expect(telemetry.dopplerRatio).toBeLessThanOrEqual(4);
expect(telemetry.dopplerRatio).toBeGreaterThanOrEqual(0.25);
expect(Number.isFinite(telemetry.dopplerCents)).toBe(true);
});
});
describe('lifecycle', () => {
it('reports live telemetry before any AudioContext exists', () => {
const engine = new AudioEngine();
const telemetry = engine.updateSpatial(poseAt(8), 1 / 60);
expect(engine.context).toBeNull();
expect(telemetry.running).toBe(false);
expect(telemetry.distance).toBeCloseTo(8, 6);
expect(telemetry.distanceGainDb).toBeLessThan(0);
expect(telemetry.criticalDistance).toBeGreaterThan(0);
expect(audio.instances).toHaveLength(0);
});
it('resumes a suspended context on start', async () => {
const engine = new AudioEngine();
await engine.start();
expect(ctxOf().state).toBe('running');
expect(engine.isPlaying).toBe(true);
});
it('is idempotent across repeated start and stop', async () => {
const engine = new AudioEngine();
await engine.start();
await engine.start();
expect(ctxOf().started).toHaveLength(1);
engine.stop();
engine.stop();
expect(engine.isPlaying).toBe(false);
await engine.start();
expect(engine.isPlaying).toBe(true);
expect(ctxOf().started).toHaveLength(2);
});
it('swaps presets without tearing down the graph', async () => {
const engine = new AudioEngine();
await engine.start();
const edgeCount = ctxOf().connections.length;
engine.update({ preset: 'engine' });
engine.update({ preset: 'beacon' });
expect(engine.getSettings().preset).toBe('beacon');
// A new source node reconnects; the rest of the graph must be untouched.
expect(ctxOf().connections.length).toBeLessThanOrEqual(edgeCount + 4);
});
it('rebuilds room acoustics when the geometry changes', async () => {
const engine = new AudioEngine();
await engine.start();
const before = engine.getAcoustics();
engine.update({ room: { width: 40, height: 16, depth: 40 }, surface: 'cathedral' });
const after = engine.getAcoustics();
expect(after.t60).toBeGreaterThan(before.t60);
expect(after.criticalDistance).toBeLessThan(before.criticalDistance);
});
it('merges partial room patches instead of dropping dimensions', () => {
const engine = new AudioEngine();
engine.update({ room: { width: 30 } as never });
const room = engine.getSettings().room;
expect(room.width).toBe(30);
expect(room.height).toBe(DEFAULT_SETTINGS.room.height);
expect(room.depth).toBe(DEFAULT_SETTINGS.room.depth);
});
it('clamps master volume', () => {
const engine = new AudioEngine();
engine.setMasterVolume(5);
expect(engine.getSettings().masterVolume).toBe(1);
engine.setMasterVolume(-2);
expect(engine.getSettings().masterVolume).toBe(0);
});
it('produces finite telemetry for degenerate geometry', () => {
const engine = new AudioEngine();
const coincident = engine.updateSpatial(
{
sourcePos: { x: 1, y: 1, z: 1 },
sourceForward: { x: 0, y: 0, z: 0 },
listenerPos: { x: 1, y: 1, z: 1 },
listenerForward: { x: 0, y: 0, z: 0 },
listenerUp: { x: 0, y: 0, z: 0 },
occlusion: 0,
},
0
);
for (const value of Object.values(coincident)) {
if (typeof value === 'number') expect(Number.isFinite(value)).toBe(true);
}
});
});
describe('telemetry consistency', () => {
it('sums the stage losses into the total, exactly', async () => {
const engine = new AudioEngine();
await engine.start();
const t = engine.updateSpatial(
poseAt(0, -7, { occlusion: 0.6, sourceForward: { x: 1, y: 0, z: 0 } }),
1 / 60
);
expect(t.distanceGainDb + t.coneGainDb + t.occlusionGainDb).toBeCloseTo(t.directGainDb, 6);
});
it('derives the direct-to-reverb ratio from the two levels it reports', async () => {
const engine = new AudioEngine();
await engine.start();
const t = engine.updateSpatial(poseAt(9), 1 / 60);
expect(t.directToReverbDb).toBeCloseTo(t.directGainDb - t.reverbGainDb, 6);
});
it('flags room dominance at the critical distance when the listener is on-axis', async () => {
const engine = new AudioEngine();
await engine.start();
const rc = engine.getAcoustics().criticalDistance;
// poseAt(0, -d) puts the listener dead ahead of a source facing −Z, so the
// cone contributes no attenuation and distance alone decides.
expect(engine.updateSpatial(poseAt(0, -rc * 0.5), 1 / 60).reverbDominant).toBe(false);
expect(engine.updateSpatial(poseAt(0, -rc * 2), 1 / 60).reverbDominant).toBe(true);
});
it('counts directivity towards room dominance, not just distance', async () => {
const engine = new AudioEngine();
await engine.start();
const rc = engine.getAcoustics().criticalDistance;
// Well inside the critical distance, but aimed away: the direct sound loses
// to the reverberant field even though the source is close.
const behind = engine.updateSpatial(poseAt(0, rc * 0.5), 1 / 60);
expect(behind.distance).toBeLessThan(rc);
expect(behind.reverbDominant).toBe(true);
});
});