Every failure below actually happened to this project and is recorded in the repository’s own logs — none are hypothetical, and each entry says plainly whether it is fixed or still live.
📖 How to read this log
Entries are newest first. The Status column is the important one:
| Status | Meaning |
|---|---|
| 🔴 Live | Still reproducible today. Cross-referenced to its blocker on 🗺️ Roadmap |
| ✅ Fixed | Confirmed resolved in current source |
| 📜 Historical | Resolved long ago; kept because the lesson or the safeguard it produced still matters |
Two failures recorded in the repository’s raw error log carry no timestamp in the log itself, so they appear last rather than being given an invented date.
🧭 Current source findings
These entries distinguish defects found during the 2026-08-02 wiki audit (B13–B16) from B17, found during the 2026-08-03 closure-plan review. They are not the same as historical runtime incidents.
| Symptom | Root cause | Status |
|---|---|---|
ModuleNotFoundError: No module named 'nltk' before generation starts | An optional dependency imported at module scope, shipped only in the local and full extras | ✅ Resolved at the optional-import boundary — B1 |
| Timeline generation reports success while the agentic path never executes | An undefined logger raises, and the deterministic fallback absorbs the error | 🔴 Live — B3 |
TypeError: Can't instantiate abstract class MockImageService | An abstract method left unimplemented on the mock | 🔴 Live — B4 |
BadRequestError: LLM Provider NOT provided from the image-prompt crew | A mock sentinel model string handed to litellm | 🔴 Live — B5 |
AttributeError: 'coroutine' object has no attribute 'id' | An async conversion never propagated to call sites | 🔴 Live — B6 |
| API dependencies, Web settings, and config-loader stdout noise | Discovered during the 2026-08-02 wiki audit; repaired in focused source work | ✅ Resolved at named boundaries — B13, B14, B15, B16 |
| Status endpoint required story_id and returned the wrong response field | Discovered during the 2026-08-03 closure-plan review, after the B1–B16 audit | ✅ Resolved at the response boundary — B17 |
🔊 2026-02-12 — the SFX cascade collapse
Symptom. Every sound effect resolved to silence. The provider cascade crashed outright, and German prompts fell into an English-only keyword path that matched nothing.
Root cause. The mock SFX provider did not satisfy the provider contract — it was missing the provider-name attribute the cascade required — so the failure of the mock took down the entire chain. Provider ordering and the keyword and configuration mappings were independently incomplete.
Fix. An eleven-phase overhaul: the wiring moved from a selector to an orchestrator, provider ordering was corrected for the ElevenLabs strategy, SFX configuration parameters were actually applied, and the keyword maps were substantially expanded.
Lesson. A mock that does not satisfy the same contract as the real implementation is not a safety net — it is an extra failure mode. Exactly this class of defect is live again today as B4.
💥 2025-10-12 — total generation failure from three missing imports
Symptom. 100% of story generations failed with name 'generate_character_id' is not defined. The pipeline stopped at stage two of three in the then-current layout. Production was fully blocked.
Root cause. A refactor extracted the character-ID helper into a shared utility but omitted the import in three consumers: the audio-production service, the character-mapping service, and the crew transformer service.
Fix. Three import statements across three files. Discovery to verified fix took 38 minutes; the integration suite went to 11 of 11 passing and generation was restored.
Lesson. This is the incident that motivated the project’s integration suite as a regression net. It is also the archetype of the undefined-name family still open today as B10 — the same mistake, caught by tests here, uncaught there.
🎙️ 2025-10-12 — voice and audio polish
Symptom. Configuration layering for the Piper high-quality-only preference behaved inconsistently; ambient padding was applied too broadly; Midjourney upsample filenames were irregular; the default voice fallback resolved to the wrong identity field.
Fix. Configuration layering hardened, ambient padding scoped to music and ambient SFX only, upsample filenames normalized, and the default voice fallback corrected to use the voice identity field.
Status. 📜 Historical.
📈 2025-10-03 — observability built after flying blind
Symptom. Diagnosing pipeline behaviour required guesswork; there was no per-stage timing or failure attribution, and storage growth was unexplained.
Root cause. No instrumentation existed. Forensic review also found substantial reclaimable storage — 8,792 compiled Python files were purged.
Fix. OpenTelemetry instrumentation across seven of the nine major stages, followed the same day by metrics instrumentation of the helper stages: character mapping, environment setup, image prompts, story plan, story writing, timeline generation and direct audio.
Lesson. This is the origin of everything on 📈 Observability. The instrumentation was added because an outage could not be diagnosed without it.
🧟 2025-09-30 — the zombie process
Symptom. Generation appeared frozen for more than fifty minutes. The process was still visible in the process table, so it looked alive. It had in fact died after about one minute.
Root cause. Two compounding problems. The child process crashed while the parent shell kept the process entry visible, hiding the death. And the crash itself came from an exception handler that referenced an undefined start_time variable — so the code meant to report the error raised its own error instead. A container constructor mismatch contributed.
Fix. A four-layer timeout defence, corrected container wiring, and — landing hours later the same night — heartbeat logging that emits every ten seconds, plus fixes to three variable-scope bugs and a missing return value in the audio-production service.
Lesson. The one that stuck: never trust a silent process. A long-running pipeline must prove liveness continuously rather than only reporting at completion, because the failure mode of silence is indistinguishable from the failure mode of work. Heartbeat logging exists in this codebase because of this night.
A second lesson, still unlearned. An exception handler that can itself raise converts a diagnosable failure into a mystery. That is structurally identical to B3, where a logging call inside an error path raises and the fallback swallows it.
📁 2025-09-12 to 2025-09-13 — artifact chaos
Symptom. Generated files landed in inconsistent locations with cryptic identifier-based names, making outputs hard to inspect and timeline visualizations hard to find.
Fix. The hierarchical artifact layout still in use today — audio split by kind, images under a visual directory, text artifacts together, all inside a timestamped, title-named run directory — plus semantic filenames derived from the story title instead of raw identifiers, and a full-name timeline legend with scene separators.
Status. 📜 Historical. The layout it produced is documented on 🏃 Runbook.
⛔ 2025-08-27 — SFX cancellation instability
Symptom. The SFX pipeline raised cancellation errors under load and could hang without a bound.
Root cause. Insufficient cancellation handling and missing timeouts in SFX orchestration.
Fix. Explicit cancellation handling, richer logging, and sensible timeouts; the local-only SFX path was validated afterwards.
Status. 📜 Historical.
🔇 2025-07-29 — silent SFX and music in montages
Symptom. Completed stories rendered with narration and dialogue but silent sound effects and background music.
Root cause. Enum values were not serialized correctly on the way into montage assembly.
Fix. Corrected enum serialization. A related montage serialization defect recurred later and is now fixed independently — see the undated log entries below.
Status. 📜 Historical.
🧪 2025-07-20 — corrupted environment
Symptom. The project reached an unstable state that blocked development.
Root cause. Recorded as environment corruption; the repository’s own record gives no narrower technical cause, and none has been reconstructed here rather than guessed.
Status. 📜 Historical.
📋 Undated entries from the raw error log
The repository keeps a raw error log at its root. It captures two failures verbatim but carries no reliable timestamps, so no date is asserted for either.
| Symptom | Root cause | Status |
|---|---|---|
litellm.BadRequestError: LLM Provider NOT provided, with the model reported as a mock sentinel | The mock sentinel string reaches litellm, which has no provider to route it to | 🔴 Live — B5 |
| Montage assembly cannot serialize a narration segment into its inputs file | A Pydantic model reached JSON serialization without JSON-mode conversion | ✅ Fixed — montage serialization now uses JSON mode |
🧭 Patterns worth naming
Reading the whole log, the same three failure shapes recur:
| Pattern | Appearances | Why it keeps happening |
|---|---|---|
| A name that was never imported | 2025-10-12, the 2025-09-30 exception handler, and live today as B2, B3, B9, B10 | Deferred annotation evaluation and broad exception handling both let undefined names survive until runtime |
| A fallback that hides the failure it caught | 2025-09-30, and live today as B3 | A fallback that does not report why it engaged converts a loud bug into a silent one |
| A mock that does not honour the real contract | 2026-02-12, and live today as B4 and B5 | Mocks are written to satisfy one call site rather than the interface |
Every one of those three patterns has an open instance today. That is the most useful thing this log says.
🔗 Related
- 🚑 Troubleshooting — what to do when you hit one of these now
- 🗺️ Roadmap — the live blockers these incidents keep pointing at
- 🏛️ Decision Log — the durable choices several of these incidents forced