📈 Observability

Observability answers which metrics, traces, and logs expose StoryMatrix pipeline and service health.

📊 Prometheus metrics

The metrics module registers 19 objects under the storymatrix_ namespace: seven counters, six histograms, five gauges, and one Info metric (src/storymatrix/infrastructure/observability/metrics.py).

Name	Type	Labels	Meaning
storymatrix_stories_generated_total	Counter	status	Completed story-generation outcomes.
storymatrix_stage_failures_total	Counter	stage, error_type	Pipeline stage failures by stage and exception type.
storymatrix_api_requests_total	Counter	method, endpoint, status_code	API request totals.
storymatrix_tts_requests_total	Counter	provider, status	TTS synthesis requests.
storymatrix_sfx_requests_total	Counter	provider, status	SFX generation requests.
storymatrix_music_requests_total	Counter	provider, status	Music generation requests.
storymatrix_llm_requests_total	Counter	provider, model, status	LLM API requests.
storymatrix_stage_duration_seconds	Histogram	stage	Pipeline stage durations.
storymatrix_generation_time_seconds	Histogram	scenes, characters	End-to-end generation duration by request shape.
storymatrix_tts_duration_seconds	Histogram	provider	TTS synthesis duration.
storymatrix_audio_length_seconds	Histogram	type	Generated audio length by audio type.
storymatrix_llm_response_time_seconds	Histogram	provider, model	LLM response latency.
storymatrix_file_size_bytes	Histogram	type	Generated file sizes by file type.
storymatrix_active_generations	Gauge	none	Active story generations.
storymatrix_pipeline_queue_size	Gauge	none	Stories waiting in the pipeline queue.
storymatrix_voice_cache_size	Gauge	none	Voices in the cache.
storymatrix_asset_library_size	Gauge	type	Assets in the library by type.
storymatrix_memory_usage_bytes	Gauge	component	Current memory use by component.
storymatrix_info	Info	version, python_version, environment	StoryMatrix version and runtime configuration information.

The Info object is declared as storymatrix and Prometheus exposes its storymatrix_info family (src/storymatrix/infrastructure/observability/metrics.py).

🔭 OpenTelemetry tracing

init_tracing() creates a resource with a fixed service name supplied by each entry point (storymatrix-api for the API and storymatrix-web for the web UI), plus the deployment environment, then installs a global tracer provider (src/storymatrix/infrastructure/observability/tracing.py, src/storymatrix/interfaces/api/main.py, src/storymatrix/interfaces/web/main.py). In the API and web startup paths, console export is off by default because each passes the OTEL_CONSOLE_EXPORTER default false; the lower-level tracing helper itself defaults to console export on when called directly. OTLP spans use gRPC when enable_otlp is true and an endpoint is available.

The runtime handles these telemetry settings as follows:

Variable	Consumer	Role
OTEL_EXPORTER_OTLP_ENDPOINT	Read by application code in the tracing module	OTLP collector endpoint when OTLP export is enabled.
OTEL_SERVICE_NAME	SDK-level convention; not set or read by StoryMatrix application code	The OpenTelemetry convention for service identity, superseded here by each entry point’s fixed service name.
OTEL_RESOURCE_ATTRIBUTES	SDK-level convention; not set or read by StoryMatrix application code	The OpenTelemetry convention for additional resource attributes; StoryMatrix supplies its own fixed resource fields.
OTEL_CONSOLE_EXPORTER	Read by application code in API and web startup	Enables console export only when set to true; the effective startup default is false.
DEPLOYMENT_ENVIRONMENT	Read by application code in API and web startup	Deployment environment passed to tracing resource metadata; defaults to development.

trace_pipeline_stage(stage_name, story_id) decorates an async stage, creates a span, records stage.name and optional story.id, marks success, and records exception type and message before re-raising failures (src/storymatrix/infrastructure/observability/tracing.py).

🪵 Loguru files

configure_logging(log_name="app") creates the configured log directory and installs console, main-file, and debug-file sinks (src/storymatrix/config/logging.py). The default files are app.log and app_debug.log under config.app.log_dir.

File	Level	Rotation	Retention	Compression
app.log	Configured application level	10 MB	7 days	zip
app_debug.log	DEBUG	25 MB	3 days	zip

Both file sinks enqueue writes and enable backtraces and diagnostics (src/storymatrix/config/logging.py).

🌐 Metrics scrape endpoint

FastAPI serves Prometheus output at GET /metrics (src/storymatrix/interfaces/api/main.py). Scrape this endpoint from the API service and correlate request counters with stage histograms and logs.