StoryMatrix turns timed narration, dialogue, effects, and music segments into a mastered montage and separate audio stems.
🔗 Chain
flowchart LR TTS["TTS\ntext segments"] --> SFX["SFX\nimpact + ambient"] SFX --> MUSIC["Music\nscene tracks"] MUSIC --> MIX["Mix\nlayered buses + ducking"] MIX --> MASTER["Master\nloudness + compression"] MASTER --> OUT["Montage + WAV stems"]
🗣️ TTS
Narration and dialogue segments carry text, timing, speaker information, and an audio source. The montage service maps both segment types into the tts bus, trims each source to its timeline slot, applies any segment gain, delays it to start_ms, and mixes the bus with the other layers (infrastructure/services/ffmpeg_montage.py). The TTS bus supplies the sidechain key for music ducking when both TTS and music exist.
💥 SFX
SFX segments enter the sfx bus. Their stem_type separates impact_sfx from ambient_sfx; when it is absent, stem selection includes the segment in both the impact_sfx and ambient_sfx lists (infrastructure/services/ffmpeg_montage.py). Ambient SFX and music can pad short sources to their timeline slots through mixing.pad_short_ambient. The SFX bus joins TTS and the ducked music bus in the final mix.
🎼 Music
Music segments enter the music bus. Each source is trimmed or looped to its target slot, receives optional soft in/out fades from mixing.music_fade_ms, and is delayed to its timeline position. When SERVICES__MONTAGE__MUSIC_CROSSFADE_ENABLED enables scene crossfades and more than one music stream exists, the service detects adjacent scene boundaries and joins those streams with FFmpeg acrossfade. The duration comes from SERVICES__MONTAGE__MUSIC_CROSSFADE_MS; otherwise the streams use amix.
🎚️ Mixing profiles
MixingProfiles exposes standard, dialogue_focused, and cinematic. Each profile supplies a target level and a ducking amount for narration, dialogue, music, SFX, and transitions (config/models.py:MixingProfiles).
| Profile | Segment type | Target | Ducking amount |
|---|---|---|---|
standard | narration | -18.0 dBFS | 0 dB |
standard | dialogue | -18.0 dBFS | 0 dB |
standard | music | -28.0 dBFS | -18.0 dB |
standard | sfx | -24.0 dBFS | -15.0 dB |
standard | transition | -24.0 dBFS | -15.0 dB |
dialogue_focused | narration | -20.0 dBFS | -2 dB |
dialogue_focused | dialogue | -16.0 dBFS | 0 dB |
dialogue_focused | music | -32.0 dBFS | -22.0 dB |
dialogue_focused | sfx | -28.0 dBFS | -18.0 dB |
dialogue_focused | transition | -26.0 dBFS | -16.0 dB |
cinematic | narration | -22.0 dBFS | -4 dB |
cinematic | dialogue | -20.0 dBFS | -2 dB |
cinematic | music | -26.0 dBFS | -20.0 dB |
cinematic | sfx | -22.0 dBFS | -14.0 dB |
cinematic | transition | -20.0 dBFS | -12.0 dB |
MixingProfileSettings stores target_level_dbfs and ducking_amount_db; get_profile() returns the named profile or standard when the name is unknown (config/models.py:MixingProfileSettings). The montage service applies each target as a gain for the corresponding segment type, then overlays foreground (narration, dialogue, sfx) and background (music, ambient) layers (infrastructure/services/montage.py).
🦆 Ducking and crossfades
The FFmpeg path creates TTS, music, and SFX buses with amix. When ducking is enabled and both TTS and music are present, it splits the TTS bus and applies sidechaincompress to music. The filter settings are controlled by these configuration keys (infrastructure/services/ffmpeg_montage.py):
| Key | Default used by FFmpeg montage | Effect |
|---|---|---|
mixing.ducking.enabled | true | Enables sidechain compression |
mixing.ducking.threshold | 0.05 | Sidechain threshold |
mixing.ducking.ratio | 8.0 | Compression ratio |
mixing.ducking.attack_ms | 5.0 | Ducking attack |
mixing.ducking.release_ms | 200.0 | Ducking release |
mixing.music_fade_ms | 200 | Soft music fade-in and fade-out |
mixing.pad_short_ambient | true | Pads short music or ambient SFX to the slot |
music_crossfade_enabled | false | Enables scene-boundary music crossfades |
music_crossfade_ms | 1000 | acrossfade duration |
crossfade_duration_ms | 500 | Montage crossfade setting in MontageConfig |
The pydub montage path applies profile-specific ducking amounts around foreground segments, changes the affected background region with a short fade at each edge, and overlays the ducked background with the foreground (infrastructure/services/montage.py).
🎛️ Mastering
MasteringConfig defines loudness and dynamics settings: enabled is true, loudness_target is -20.0, loudness_range is 7.0, true_peak is -2.0, compressor threshold is -20.0, ratio is 4.0, attack is 0.005, release is 0.1, and limiter threshold is -1.0 (config/models.py:MasteringConfig). The FFmpeg accessor defaults mastering to disabled (false) when the setting is absent; when enabled, its fallback values are loudness -16.0, range 11.0, true peak -1.0, compressor threshold 0.5, ratio 2.0, attack 20.0, and release 100.0, rather than the model defaults. The filter graph applies only loudnorm and acompressor; limiter_threshold is config-only and unused in this path. Compressor attack is clamped to a minimum of 0.01 at runtime, so the model default 0.005 becomes 0.01 when used. The final stream is [final_mix]; disabling mastering passes the mixed stream through anull.
MontageConfig sets target_loudness_dbfs to -20.0, selects the ffmpeg implementation, and carries mastering plus crossfade settings (config/models.py:MontageConfig).
🧪 FFmpeg execution
FFmpegMontageService receives binary paths from the container’s paths.ffmpeg and paths.ffprobe values. PathsSettings defaults them to /usr/bin/ffmpeg and /usr/bin/ffprobe; the service checks each executable with shutil.which (config/models.py:PathsSettings, infrastructure/container.py, infrastructure/services/ffmpeg_montage.py).
The service invokes ffmpeg with one -i input per valid segment, a -filter_complex graph, the [final_mix] map, stereo output (-ac 2), and 44.1 kHz output (-ar 44100). Each input chain applies gain, timestamp reset, trim or loop, optional ambient padding, fades, and timeline delay. Per-track streams combine into buses with amix; music optionally uses acrossfade; sidechain compression ducks music beneath TTS; and mastering produces the final stream (infrastructure/services/ffmpeg_montage.py).
Duration probing invokes ffprobe with -show_entries format=duration, parses the numeric result, and returns 0.0 for missing or invalid media (infrastructure/services/ffmpeg_montage.py).
🧵 Four exported stems
Stem generation writes files below the output file’s artifacts/audio/stems/ directory (infrastructure/services/ffmpeg_montage.py). The four available stem names are:
| Stem | Included segments | Filename suffix |
|---|---|---|
| TTS | narration and dialogue | _tts.wav |
| Music | music | _music.wav |
| Impact effects | SFX with stem_type == "impact_sfx" | _impact_sfx.wav |
| Ambient effects | SFX with stem_type == "ambient_sfx" | _ambient_sfx.wav |
Stem profiles disable ducking and loudness normalization, and each copied segment resets volume adjustment to 0 dB so the exported files remain unmixed (infrastructure/services/ffmpeg_montage.py). The ambient_sfx stem is available and exports as the fourth stem (application/production/stages/finalize_production.py).