🎞️ Production Pipeline

The production pipeline answers how StoryMatrix moves a generation request through nine ordered stages and resumes after a checkpoint.

🧭 Fixed stage order

GenerateStoryUseCase constructs StoryProductionPipeline with this exact order (src/storymatrix/application/use_cases/generate_story.py):

#	Stage class	Implementation	Reads from context	Writes to context	Skip flag or condition
1	CharacterMappingStage	application/production/stages/character_mapping_stage.py	request, optional story_plan, provider configuration	character_map	None
2	StoryPlanStage	application/production/stages/story_plan_stage.py	request, character_map, temp_path	story_plan, story_data, and a refreshed character_map when new names appear	None
3	StoryWritingStage	application/production/stages/story_writing_stage.py	output_path, request, story_plan, story_data, character_map	story	STORY.md at output_path selects author-provided ingestion
4	ImagePromptGenerationStage	application/production/stages/image_prompt_generation_stage.py	story, story_plan, story_data	enhanced_image_prompts	Provider failure clears prompts and returns the context
5	GenerateMediaAssetsStage	application/production/stages/generate_media_assets.py	story, request, timeline, temp_path, output_path, enhanced_image_prompts	generated_audio_segments, artifacts, and asset_durations	request.skip_steps["image"] skips image generation; request.sfx and request.ambient_sounds select audio content
6	TimelineGenerationStage	application/production/stages/timeline_generation_stage.py	story, character_map, temp_path, request, generated_audio_segments	timeline	Requires story and character_map
7	AssembleAudioStage	application/production/stages/assemble_audio.py	story, timeline, generated_audio_segments, artifacts, request	Final audio artifacts and audio metadata	Missing timeline or audio segments returns the context without assembly
8	FinalizeProductionStage	application/production/stages/finalize_production.py	story, final-audio artifacts, output_path, artifacts_path	Master files, stem metadata, and final output metadata	Missing final-audio artifact returns the context
9	CleanupStage	application/production/stages/cleanup_stage.py	temp_path, artifacts_path, request.keep_temp_files	Filesystem cleanup state	keep_temp_files and path-safety checks skip deletion

🔗 Execution flow

flowchart LR
    A[CharacterMappingStage] --> B[StoryPlanStage]
    B --> C[StoryWritingStage]
    C --> D[ImagePromptGenerationStage]
    D --> E[GenerateMediaAssetsStage]
    E --> F[TimelineGenerationStage]
    F --> G[AssembleAudioStage]
    G --> H[FinalizeProductionStage]
    H --> I[CleanupStage]

📦 StoryProductionContext

StoryProductionContext is a Pydantic BaseModel with ConfigDict(arbitrary_types_allowed=True), allowing domain objects and filesystem paths to pass between stages (src/storymatrix/application/production/context.py).

Field	Type	Role
request	GenerateStoryRequest	Original generation options
story	Story | None	Domain story aggregate
story_plan	StoryPlan | None	Crew-generated planning result
story_data	StoryData | None	Structured story data
timeline	AgenticTimeline | None	Audio timeline
generated_audio_segments	list[dict[str, Any]] | None	Generated audio segment records
artifacts_path	Path	Artifact working directory
temp_path	Path	Temporary working directory
output_path	Path | None	Output directory
generation_timestamp	float | None	Generation start timestamp
characters	list[Character] | None	Character collection
character_map	dict | None	Character-to-voice mapping
enhanced_image_prompts	list[dict[str, str]] | None	Crew-enhanced image prompts
metadata	dict[str, Any]	Arbitrary stage metadata
artifacts	list[Artifact]	Produced artifact records
asset_durations	dict[str, float]	Duration metadata keyed by asset
completed_stages	list[str]	Completed stage class names
stage_checkpoints	dict[str, datetime]	Completion timestamps by stage
pipeline_state	dict[str, Any]	Additional resumability state

💾 Checkpointing and resume

Checkpointing is enabled by default on StoryProductionPipeline. After each stage returns a non-None context, the pipeline marks the stage class name complete and writes <artifacts_path>/checkpoint.json; if a stage raises, the exception path writes the current context checkpoint before re-raising (src/storymatrix/application/production/pipeline.py).

The JSON checkpoint contains completed_stages, ISO-formatted stage_checkpoints, pipeline_state, generation_timestamp, artifacts_count, and metadata (src/storymatrix/application/production/context.py). Loading restores completed stage names, parses checkpoint timestamps, restores pipeline state, and merges saved metadata.

--resume reaches the use case request as request.resume and is passed to StoryProductionPipeline.run(resume=...) (src/storymatrix/application/use_cases/generate_story.py). When resume is true, the pipeline attempts to load the checkpoint, computes get_next_stages(all_stage_names) when the context already has completed stages, and logs that result; the dispatch loop does not consume that computed list. Instead, it always iterates the original nine-stage list in order and skips each stage whose class name is already present in completed_stages via is_stage_completed; the first absent class name is therefore the next stage that runs (src/storymatrix/application/production/pipeline.py, src/storymatrix/application/production/context.py). If checkpoint loading fails or no checkpoint exists on a fresh context, completed_stages remains at its fresh default of an empty list, so execution starts at CharacterMappingStage; loading does not clear an existing completed-stage set (src/storymatrix/application/production/context.py).

🗂️ Output contract

The output root is out/, selected by config/models.py:AppSettings.output_dir. Finalization organizes the master output and artifacts below that root (src/storymatrix/application/production/stages/finalize_production.py). The artifact layout is read from the code because no pipeline run has completed in this checkout.

⚠️ Known break in the timeline stage

LLMAgenticTimelineAdapter.direct_audio_timeline calls logger in six response-parsing call sites, but llm_agentic_timeline_adapter.py does not import or define that symbol (src/storymatrix/infrastructure/adapters/timeline/llm_agentic_timeline_adapter.py). The resulting NameError enters the deterministic-timeline fallback, which absorbs the failure. The agentic response path therefore never runs successfully while the timeline stage still returns the deterministic fallback as if successful. The fix is to import the project logger before those calls. Track this defect at index > b3.