ai_synth/docs/audit/v2-qa-report.md

AI Weekly Synth -- Test Coverage Audit Report (v2)

Date: 2026-03-27 Auditor: QA Engineer (automated analysis)

---

1. Test Inventory

1.1 Backend Unit Tests (cargo test --lib)

Total: 358 tests -- all passing.

Source file	# tests	Coverage area
services/scraper.rs	74	SSRF IP checks, soft-404, redirect, HTML parsing
services/synthesis.rs	36	Pipeline logic, schema building, category overflow
services/llm/anthropic.rs	20	Response parsing, error handling
services/prompts.rs	18	Prompt template generation
services/csv.rs	18	CSV parsing, serialisation
models/synthesis.rs	16	Model validation, serialisation
services/rate_limiter.rs	15	Token bucket, concurrency
services/llm/openai.rs	13	Response parsing, error handling
models/source.rs	12	URL / title validation
models/settings.rs	12	Settings validation, defaults
services/export.rs	12	Markdown / PDF rendering
services/llm/gemini.rs	10	Response parsing, error handling
models/provider.rs	10	Provider / model validation
services/email.rs	9	Email rendering, bypass mode
services/encryption.rs	8	AES-256-GCM encrypt/decrypt
services/source_scraper.rs	8	Link extraction, is_article filter
services/llm/schema.rs	8	JSON schema generation
util/token.rs	8	Token generation, hashing
models/api_key.rs	8	API key validation
middleware/csrf.rs	7	CSRF header check
models/rate_limit.rs	6	Rate limit model validation
config.rs	6	Config parsing
middleware/auth.rs	5	Session extraction
services/llm/factory.rs	5	Provider factory
handlers/admin.rs	4	Admin handler validation
services/brave_search.rs	1	Brave search (minimal)
services/llm/mock.rs	0	Mock provider (no assertions)
errors.rs	0	Error types (no unit tests)

1.2 Backend Integration Tests (backend/tests/)

Total: 183 tests across 17 files (requires Postgres).

File	# tests	Coverage area
api_sources_test.rs	36	Sources CRUD, validation, CSV, bulk import, max limit
api_admin_test.rs	30	Provider CRUD, rate limits, user mgmt, audit log, config
api_keys_test.rs	18	API key CRUD, encryption, ownership, test endpoint
api_syntheses_test.rs	17	Synthesis CRUD, pagination, ownership, generation trigger
api_auth_test.rs	16	Register, login, verify, logout, session
api_export_test.rs	13	Email send, Markdown export, PDF export
api_themes_test.rs	10	Theme CRUD, validation, ownership
api_schedules_test.rs	9	Schedule CRUD, validation, ownership
api_settings_test.rs	7	Settings CRUD, defaults, boundary, isolation
pipeline_test.rs	6	Phase 1 extraction, Phase 2 search, overflow, diversity, dedup, preferred
api_article_history_test.rs	4	History list, clear, provenance
api_csrf_test.rs	4	CSRF header enforcement
api_stop_generation_test.rs	4	Stop job, ownership, 404
api_llm_logs_test.rs	3	LLM logs auth, 404, happy path
api_sources_preferred_test.rs	3	Preferred sources set/clear/auth
minimal_test.rs	2	Infrastructure sanity (oneshot)
api_health_test.rs	1	Health check

1.3 Frontend Unit Tests (vitest)

Total: 141 tests (131 passing, 10 failing) across 18 files.

File	# tests	Coverage area
sources-utils.test.ts	20	Source utilities
provider-info.test.ts	11	Provider info helpers
api-keys.test.ts	11	API keys client
synthesis-utils.test.ts	11	Synthesis utilities
sse.test.ts	11	SSE client
settings-validation.test.ts	3	Settings validation
i18n.test.ts	9	i18n translations
api-client.test.ts	7	API client
config-api.test.ts	7	Config API
pages/settings.test.tsx	10	Settings page
pages/sources.test.tsx	8	Sources page
pages/home.test.tsx	7	Home page
pages/generate.test.tsx	6	Generate page
synthesis-export.test.ts	6	Export utilities
pages/login.test.tsx	4	Login page
pages/register.test.tsx	4	Register page
auth-context.test.tsx	3	Auth context
admin-route-guard.test.tsx	3	Admin route guard

1.4 E2E Tests (Playwright)

Total: 7 tests across 7 files.

File	Coverage area
registration.spec.ts	Full magic link registration flow
settings.spec.ts	Settings persistence across reloads
settings-export.spec.ts	Settings export/import roundtrip
sources.spec.ts	Source CRUD + preferred sources via API
themes.spec.ts	Theme CRUD + schedule CRUD via API
admin-providers.spec.ts	Admin provider management, settings dropdown
generation-live.spec.ts	Full pipeline with real OpenAI key (gated)

---

2. Feature Coverage Matrix

Feature	Unit Tests	Integration Tests	E2E Tests	Coverage
Auth: register	-	4 tests	1 test	GOOD
Auth: login	-	3 tests	-	GOOD
Auth: magic link verify	-	3 tests	1 test	GOOD
Auth: /me	-	3 tests	-	GOOD
Auth: logout	-	3 tests	-	GOOD
Auth: session expiry	5 (middleware)	-	-	PARTIAL
CSRF protection	7 (middleware)	4 tests	-	GOOD
Settings CRUD	12 (model)	7 tests	2 tests	GOOD
Sources CRUD	12 (model)	36 tests	1 test	GOOD
Sources: CSV import/export	18 (csv)	6 tests	-	GOOD
Sources: preferred	-	3 tests	1 test	GOOD
Sources: max limit	-	2 tests	-	GOOD
Themes CRUD	-	10 tests	1 test	GOOD
Schedules CRUD	-	9 tests	1 test	GOOD
Scheduled execution	0	0	0	NONE
Syntheses CRUD	16 (model)	11 tests	-	GOOD
Syntheses: pagination	-	2 tests	-	GOOD
Syntheses: ownership isolation	-	2 tests	-	GOOD
Generation: trigger	-	4 tests	1 test (gated)	GOOD
Generation: SSE progress	11 (sse client)	0	1 test (gated)	PARTIAL
Generation: stop	-	4 tests	0	GOOD
Pipeline: Phase 1 (scrape)	74+8 (scraper)	1 test	1 test (gated)	GOOD
Pipeline: Phase 2 (search)	-	1 test	1 test (gated)	GOOD
Pipeline: category overflow	36 (synthesis)	1 test	-	GOOD
Pipeline: is_article filter	8 (source_scraper)	0	0	PARTIAL
Pipeline: summary_length	18 (prompts)	0	0	PARTIAL
Pipeline: date extraction	0	0	0	NONE
Pipeline: article history dedup	-	1 test	-	GOOD
Pipeline: source diversity cap	-	1 test	1 test (gated)	GOOD
Pipeline: preferred ordering	-	1 test	-	GOOD
Pipeline: Brave Search	1 (minimal)	0	0	WEAK
API keys CRUD	8 (model)	18 tests	-	GOOD
API keys: encryption at rest	8 (encryption)	1 test	-	GOOD
API keys: test endpoint	-	2 tests	-	GOOD
Admin: provider CRUD	10 (model) + 4 (handler)	9 tests	1 test	GOOD
Admin: rate limits	6 (model) + 15 (limiter)	4 tests	-	GOOD
Admin: user management	-	6 tests	-	GOOD
Admin: audit log	-	2 tests	-	GOOD
Config: providers	-	3 tests	-	GOOD
Export: Markdown	12 (export)	4 tests	-	GOOD
Export: PDF	12 (export)	4 tests	-	GOOD
Export: Email	9 (email)	5 tests	-	GOOD
SSRF protection	74 (scraper)	0	0	PARTIAL
LLM call logging	-	3 tests	1 test (gated)	GOOD
LLM providers (Gemini)	10	-	-	GOOD
LLM providers (OpenAI)	13	-	-	GOOD
LLM providers (Anthropic)	20	-	-	GOOD
Rate limiting	15	0	0	PARTIAL
Turnstile captcha	bypass only	bypass only	bypass only	PARTIAL

---

3. Coverage Gaps and Recommendations

GAP-01: Scheduled Execution (scheduler.rs) -- No tests

Priority: Critical

The run_scheduled_jobs() function in services/scheduler.rs has zero unit tests and zero integration tests. This is a critical autonomous process that triggers generation and sends emails without user interaction.

Tests to write:

1. Unit test (scheduler.rs): run_scheduled_jobs triggers generation for themes whose schedule matches the current day+time.
2. Unit test (scheduler.rs): run_scheduled_jobs does NOT trigger generation for disabled schedules.
3. Unit test (scheduler.rs): run_scheduled_jobs does NOT trigger generation when the current day is not in the schedule's days list.
4. Integration test (api_schedules_test.rs): Create a schedule set to "now", verify the scheduler picks it up and a synthesis is created (or at least attempted with a mock provider).
5. Integration test: Verify that after run_scheduled_jobs executes, the schedule's last_run_at timestamp is updated.

---

GAP-02: SSE Progress Stream -- No integration test

Priority: High

The SSE progress endpoint (GET /api/v1/syntheses/generate/:job_id/progress) is only tested in the E2E suite (gated behind a real API key). There is no integration test that verifies the SSE connection, event format, or error propagation.

Tests to write:

1. Integration test (api_syntheses_test.rs): Connect to SSE endpoint for a running job and verify the stream sends well-formed events (progress, complete, or error).
2. Integration test: SSE endpoint returns 404 for a non-existent job_id.
3. Integration test: SSE endpoint returns 401 without auth.

---

GAP-03: Brave Search Pipeline -- Minimal coverage

Priority: High

The services/brave_search.rs has only 1 unit test. The Brave Search path in the pipeline (use_brave_search: true) is explicitly commented as untestable in pipeline_test.rs because it requires a real API key. The entire search-via-Brave code path is unverified in integration tests.

Tests to write:

1. Unit test (brave_search.rs): Parse a valid Brave Search API response and extract URLs.
2. Unit test (brave_search.rs): Handle Brave API error responses (429 rate limit, 401 invalid key).
3. Unit test (brave_search.rs): Handle malformed Brave API JSON response gracefully.
4. Integration test (pipeline_test.rs): Use wiremock to mock the Brave API endpoint and run the pipeline with use_brave_search: true, verifying that Brave results feed into the pipeline.

---

GAP-04: Date Extraction / max_age_days Filtering -- No tests

Priority: High

The max_age_days field is on themes and is used to filter old articles. However, there are no tests (unit or integration) that verify articles older than max_age_days are excluded from the synthesis. The prompt includes date extraction instructions, but there is no test that validates article age filtering actually works.

Tests to write:

1. Unit test (synthesis.rs): Articles with a published_at date older than max_age_days are excluded.
2. Integration test (pipeline_test.rs): Set max_age_days: 1 and provide wiremock articles with old dates, verify they are filtered out.
3. Unit test (prompts.rs): Verify the date extraction instruction appears in the prompt when max_age_days > 0.

---

GAP-05: Rate Limiting -- No integration test

Priority: High

The rate limiter has 15 unit tests but zero integration tests. There is no test that verifies the rate limiter actually blocks LLM calls when the limit is exceeded in a real pipeline run, nor any test that the admin-configured rate limits are loaded and applied.

Tests to write:

1. Integration test: Configure a very low rate limit (e.g., max_requests: 1, time_window_seconds: 60), trigger generation with multiple sources, and verify the rate limiter introduces delays (or that the pipeline logs rate-limit waits).
2. Integration test: Verify that user-level rate limit overrides (from settings rate_limit_max_requests) are applied when set.

---

GAP-06: SSRF Protection -- No integration test

Priority: Medium

The SSRF IP checks have 74 unit tests in scraper.rs (excellent is_private_ip coverage), but there is no integration test that verifies the full check_ssrf function actually blocks a request to a private IP through the scraper pipeline. The tests bypass SSRF with SKIP_SSRF_CHECK=1.

Tests to write:

1. Integration test (pipeline_test.rs): Add a source with a URL pointing to 127.0.0.1 (without SKIP_SSRF_CHECK), verify the scraper rejects it and the pipeline continues with other sources.
2. Integration test: Verify that redirect to a private IP is blocked (source URL redirects to http://192.168.1.1).

---

GAP-07: Pipeline is_article Filter -- No end-to-end verification

Priority: Medium

The is_article heuristic in source_scraper.rs has 8 unit tests, but there is no integration test that verifies non-article pages (e.g., category index pages, about pages) are actually filtered out during a pipeline run.

Tests to write:

1. Integration test (pipeline_test.rs): Set up wiremock with a source page linking to both article pages and non-article pages (e.g., /about, /contact, /category), verify only articles make it into the synthesis.

---

GAP-08: Pipeline summary_length -- No integration test

Priority: Medium

The summary_length field on themes controls the number of sentences in generated summaries. The prompts unit tests verify the instruction appears in the prompt, but no test verifies the LLM response is actually constrained to the requested length.

Tests to write:

1. Integration test (pipeline_test.rs): Use MockLlmProvider with summary_length: 1 and verify the mock generates 1-sentence summaries (or verify the prompt includes the correct summary_length instruction).

---

GAP-09: Settings Validation Rejections -- No negative tests

Priority: Medium

The settings integration tests cover boundary values but do not test rejection of out-of-range values. There are no tests for max_articles_per_source: 0, batch_size: -1, max_links_per_source: 999, etc.

Tests to write:

1. Integration test (api_settings_test.rs): PUT /settings with max_articles_per_source: 0 returns 422.
2. Integration test (api_settings_test.rs): PUT /settings with batch_size: 0 returns 422.
3. Integration test (api_settings_test.rs): PUT /settings with max_links_per_source: 999 returns 422.
4. Integration test (api_settings_test.rs): PUT /settings with article_history_days: -1 returns 422.

---

GAP-10: Synthesis Export Content with Special Characters -- No test

Priority: Low

Export tests verify structure and content-type but do not test synthesis content with special characters (accented characters, HTML entities, emoji, very long URLs). There is no test for PDF generation with such edge cases.

Tests to write:

1. Integration test (api_export_test.rs): Insert a synthesis with UTF-8 characters, URLs with query strings, and long summaries. Export as Markdown and verify content integrity.
2. Integration test (api_export_test.rs): Same synthesis exported as PDF. Verify PDF magic bytes and non-empty content.

---

GAP-11: Concurrent Generation -- No test

Priority: Low

The generate_twice_returns_error_for_second test verifies the same user cannot run two jobs. But there is no test for two different users generating simultaneously, which is the expected multi-tenant behavior.

Tests to write:

1. Integration test (api_syntheses_test.rs): Two users trigger generation simultaneously. Both should get 202. Verify they do not interfere with each other.

---

GAP-12: Frontend -- Failing tests

Priority: Medium

10 of 141 frontend unit tests are currently failing. These failures need investigation and resolution before any new coverage work.

Action:

1. Run npx vitest run and capture the 10 failing test names.
2. Investigate root cause (likely stale mocks or component changes).
3. Fix or update the failing tests.

---

GAP-13: Article History Ownership Isolation -- No test

Priority: Medium

The article history endpoint tests cover auth and empty-state, but there is no test verifying that User B cannot see User A's article history.

Tests to write:

1. Integration test (api_article_history_test.rs): User A generates a synthesis (creating history entries). User B calls GET /article-history and sees empty results.
2. Integration test (api_article_history_test.rs): User B calls DELETE /article-history and it does not affect User A's history entries.

---

GAP-14: Provenance Ownership Isolation -- No test

Priority: Medium

The provenance endpoint test only covers the 404 case. There is no test verifying that User B cannot access User A's synthesis provenance.

Tests to write:

1. Integration test (api_article_history_test.rs): User A has a synthesis with provenance. User B calls GET /syntheses/:id/provenance on User A's synthesis and gets 404.

---

GAP-15: E2E Coverage of Core Flows -- Missing

Priority: Low

Several core user journeys are only covered by integration tests, not E2E tests:

• Login flow (only registration has an E2E test)
• API key management (add, test, delete)
• Synthesis list/detail view
• Synthesis deletion
• Markdown/PDF export download
• Email send from synthesis view

Tests to write (selected):

1. E2E test: Login via magic link, verify session persists.
2. E2E test: Add an API key, verify it appears masked in the list, test it, delete it.
3. E2E test: View synthesis detail page, click export Markdown, verify download.

---

4. Summary

Overall Assessment

Metric	Value
Backend unit tests	358 (all passing)
Backend integration tests	183
Frontend unit tests	141 (10 failing)
E2E tests	7
Total test count	689
Critical gaps	1 (scheduled execution)
High-priority gaps	4 (SSE stream, Brave Search, date filtering, rate limiting integration)
Medium-priority gaps	6
Low-priority gaps	3

Strengths

• SSRF protection has the best unit test coverage in the project (74 tests) covering all private IP ranges, IPv4-mapped IPv6, and redirect blocking.
• Sources CRUD is the most thoroughly tested API endpoint (36 integration tests) including CSV import/export, bulk import, max limits, and boundary values.
• Admin module has comprehensive access control tests (access denied for non-admin, non-authenticated, each endpoint).
• Ownership isolation is consistently tested across all user-scoped endpoints (syntheses, sources, API keys, themes, schedules, stop generation).
• Pipeline tests use wiremock and MockLlmProvider to test the full generation flow without real API keys, covering scraping, classification, overflow, diversity, dedup, and preferred ordering.
• Encryption verification directly queries the database to confirm API keys are not stored in plaintext.
• Audit logging is verified by querying the audit_log table after admin actions.
• E2E generation test exercises the complete pipeline with a real OpenAI key including provenance and LLM log verification.

Weaknesses

• Scheduled execution has zero test coverage -- a critical autonomous process.
• Brave Search pipeline path is untested beyond a single unit test.
• Date extraction/age filtering has no tests at any level.
• Rate limiting is well unit-tested but has no integration verification.
• SSE progress stream has no integration test (only gated E2E).
• Frontend has 10 failing tests that need immediate attention.
• Settings validation lacks negative boundary tests (rejection of invalid values).