🧪 Testing

Testing guidance answers what each suite level covers, how tests stay offline, and where current failures cluster.

🗂️ Test taxonomy

Directory	Coverage focus
tests/unit/	Isolated contracts for domain, application, configuration, infrastructure, interfaces, CLI, and utilities.
tests/integration/	Cross-component wiring, pipeline behavior, adapters, persistence, API/web contracts, and golden-path production.
tests/smoke/	Fast SFX orchestration smoke checks.
tests/performance/	Audio-processing performance checks.
tests/scripts/	Script-level behavior.
tests/isolation/	Sandbox preflight and network-egress protection.
tests/fixtures/	Shared fixture package directories for crew, music, TTS, and CLI data.

Pytest discovers test_*.py, Test* classes, and test_* functions under tests/; pythonpath includes src (pyproject.toml).

🏷️ Pytest markers and async mode

The five configured markers are:

Marker	Meaning
slow	Tests with longer execution time; deselect with -m "not slow".
integration	Integration tests.
unit	Unit tests.
api	Tests that require API access.
asyncio	Asynchronous tests.

asyncio_mode = "strict" requires explicit async-test handling (pyproject.toml).

🧰 Fixtures

Global fixtures in tests/conftest.py strip proxy and provider credentials through the autouse sandbox_isolation_defense_in_depth fixture and provide deterministic service doubles:

Fixture	Role
mock_llm_service	MagicMock implementing LLMService.
mock_tts_service	MagicMock that creates a dummy audio file.
mock_sfx_resolver_service	MagicMock returning no SFX.
mock_music_discovery_service	MagicMock returning no music.
mock_character_mapping_service	MagicMock returning a Robo character mapping.

Integration fixtures in tests/integration/conftest.py provide mock_characters, module-scoped mock_config, mock_timeline_dict, an async mock_llm_service returning StoryPlan, StoryData, and AudioTimeline, an async mock_tts_service exporting a silent WAV, and async SFX/music discovery services.

📴 Offline mock adapters

APP__DEV_LOCAL_ONLY=true selects offline adapters through the dependency-injection container. The development mock set is (src/storymatrix/infrastructure/adapters/DEV_MOCKS_README.md):

Category	Adapter
TTS	tts/mock_tts_adapter.py → MockTTSAdapter
Music	music/mock.py → MusicMockAdapter
Timeline	timeline/mock_agentic_timeline_adapter.py → MockAgenticTimelineAdapter

The same offline branch responds to the --offline CLI flag and missing API keys. Unit and integration tests use fixtures in tests/fixtures/ rather than importing these infrastructure mocks directly. The mock layer prevents paid TTS and provider APIs from receiving standard test traffic.

🧬 Golden-path snapshots

tests/integration/test_golden_path.py compares production outputs with static references under tests/integration/snapshots/. UUIDs and timestamps receive masking in snapshot comparisons (AGENTS.md). Refresh snapshots only after an intentional, reviewed change to story, timeline, or output schemas:

uv run pytest --snapshot-update

Review the generated diff, confirm the changed behavior is intentional, and then commit the updated references. A snapshot refresh is not a substitute for fixing an unexpected regression.

📊 Coverage and isolation

The project targets coverage above 80% (AGENTS.md). CI runs tests through scripts/sandbox-launcher.sh, which invokes bubblewrap with --unshare-net. tests/isolation/sitecustomize.py adds a socket and asyncio guard: only loopback hosts (127.0.0.1, ::1, localhost) pass, while other destinations raise PermissionError. The hook also strips proxy and PROVIDERS__ environment variables. This fail-closed boundary keeps paid provider APIs unreachable from CI and removes provider credentials before test code runs.

🩺 Current suite state (measured 2026-08-02)

Across three runs measured 2026-08-02, stable results are 12 errors, 18 skipped, and 458 non-skipped tests. The pass and failure counts vary: 391–393 passed and 53–55 failed, a pass rate of roughly 85.4–85.8% of 458 non-skipped tests. A single nondeterministic test, tests/performance/test_audio_processing.py::TestAudioProcessingPerformance::test_memory_usage_stability, flips between pass and failure, so any exact pass count published for this suite reproduces only to ±1 (measured 2026-08-02). Static checks report 1160 errors from ruff check src/ tests/, including 848 autofixable errors, and 18 F821 undefined-name findings in src/ (measured 2026-08-02). The memory-stability test is a test-quality defect distinct from the four product-failure clusters below; its nondeterminism does not identify a product bug.

🔎 Why the suite is red

The failures cluster into four root causes:

Blocker	Root cause	Observable failure	Evidence and tracking
B6	FileSystemMediaAssetRepository.save is async, while its callers still use it synchronously.	AttributeError: 'coroutine' object has no attribute 'id' across 11 repository tests.	tests/unit/infrastructure/repositories/test_file_media_asset_repository.py; index > b6
B7	StoryData.background_music is required in the schema, while the golden fixture omits it.	ValidationError during tests/integration/test_golden_path.py setup.	src/storymatrix/crew/schemas.py, tests/integration/conftest.py; index > b7
B8	Container-wiring fixtures pass a string where a Path is required.	AttributeError: 'str' object has no attribute 'exists' during setup of both container-wiring tests.	tests/integration/test_container_wiring.py; index > b8
B4	MockImageService implements only part of the abstract ImageGenerationService port.	TypeError when local-only integration wiring instantiates the mock.	src/storymatrix/infrastructure/services/image/mock_image_service.py, tests/integration/test_di_gating.py; index > b4

These four clusters describe the measured failure concentration; the remaining failures and errors require separate source-level triage.