diff --git a/audits/2026-03-27/qa-integration-e2e.md b/audits/2026-03-27/qa-integration-e2e.md index c1894a0..56d3b07 100644 --- a/audits/2026-03-27/qa-integration-e2e.md +++ b/audits/2026-03-27/qa-integration-e2e.md @@ -17,6 +17,7 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs ## 3) Prioritized Findings (P0-P3) ### P0 — Scheduled execution path is effectively untested (critical requirement risk) + - Why it matters: Scheduled generation + email delivery is a core requirement. Regressions here can silently fail user deliverables. - Evidence: - Scheduler runtime logic exists in [scheduler.rs:27](/Users/oabrivard/Projects/rust/ai_synth/backend/src/services/scheduler.rs:27) through [scheduler.rs:91](/Users/oabrivard/Projects/rust/ai_synth/backend/src/services/scheduler.rs:91). @@ -25,6 +26,7 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs - Direction: Add deterministic integration tests for due schedule selection, double-run prevention (`last_run_at`), job contention behavior, and email send invocation outcomes. ### P1 — SSE progress endpoint has no deterministic integration coverage + - Why it matters: Generation UX and cancellation safety depend on SSE correctness. - Evidence: - SSE handler is implemented in [generation.rs:150](/Users/oabrivard/Projects/rust/ai_synth/backend/src/handlers/generation.rs:150). @@ -33,6 +35,7 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs - Direction: Add integration tests that subscribe to `/progress`, assert `progress -> complete/error` sequence, ownership enforcement, reconnect semantics, and keepalive stability. ### P1 — Brave Search fallback path lacks integration coverage + - Why it matters: Fallback branch is a key functional path and currently high regression risk. - Evidence: - Brave branch in pipeline code: [synthesis.rs:371](/Users/oabrivard/Projects/rust/ai_synth/backend/src/services/synthesis.rs:371). @@ -40,6 +43,7 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs - Direction: Add mock HTTP server + encrypted Brave key fixture flow to execute `use_brave_search=true` end-to-end in integration tests. ### P1 — Pipeline integration does not verify rate-limit behavior + - Why it matters: Rate limiting is a non-functional requirement; failures can produce outages or provider bans. - Evidence: - Pipeline tests set user rate limit fields to `null` ([pipeline_test.rs:64](/Users/oabrivard/Projects/rust/ai_synth/backend/tests/pipeline_test.rs:64)). @@ -47,6 +51,7 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs - Direction: Add integration scenarios for strict user/provider limits and verify wait/retry/timeout outcomes. ### P1 — Pipeline integration does not verify max-age article filtering behavior + - Why it matters: Freshness is a core content-quality requirement. - Evidence: - Pipeline tests consistently use high `max_age_days` values ([pipeline_test.rs:77](/Users/oabrivard/Projects/rust/ai_synth/backend/tests/pipeline_test.rs:77)). @@ -54,18 +59,21 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs - Direction: Add wiremock articles with old publish dates + assertions on filtering and history status. ### P2 — E2E suite is heavily API-driven, limited UI journey validation + - Why it matters: UI regressions can pass E2E while backend endpoints stay healthy. - Evidence: - Sources and themes E2E use `page.evaluate(fetch(...))` for most operations ([sources.spec.ts:23](/Users/oabrivard/Projects/rust/ai_synth/e2e/tests/sources.spec.ts:23), [themes.spec.ts:29](/Users/oabrivard/Projects/rust/ai_synth/e2e/tests/themes.spec.ts:29)). - Direction: Keep API-assisted setup, but assert critical user interactions through UI (form submit, validation messages, control states). ### P2 — Article history ownership isolation is not explicitly tested + - Why it matters: Multi-user data isolation is security-sensitive. - Evidence: - Current article history integration tests cover auth + empty/clear/provenance 404 only ([api_article_history_test.rs:24](/Users/oabrivard/Projects/rust/ai_synth/backend/tests/api_article_history_test.rs:24)). - Direction: Add user A vs user B cross-access tests for history and provenance endpoints. ### P2 — QA guidelines are out of sync with current codebase signals + - Why it matters: stale test inventory causes false confidence in planning and release gates. - Evidence: - Documented counts/status in [qa_guidelines.md:7](/Users/oabrivard/Projects/rust/ai_synth/docs/qa_guidelines.md:7) to [qa_guidelines.md:11](/Users/oabrivard/Projects/rust/ai_synth/docs/qa_guidelines.md:11). @@ -73,6 +81,7 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs - Direction: Automate inventory generation in CI and update `docs/qa_guidelines.md` from machine output. ### P3 — Frontend unit test execution environment is currently brittle + - Why it matters: slows QA feedback loop and hides regressions. - Evidence: - Local run `cd frontend && npx vitest run` failed due missing optional Rollup binary (`@rollup/rollup-darwin-x64`). @@ -81,7 +90,7 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs ## 4) Coverage Map (Required Capability vs Current Coverage) | Capability | Unit | Integration | E2E | Status | -|---|---|---|---|---| +| --- | --- | --- | --- | --- | | Auth (register/login/verify/session) | Medium | Strong (`api_auth_test.rs`) | Medium (`registration.spec.ts`) | Good | | Theme CRUD | Low | Strong (`api_themes_test.rs`) | Medium (API-driven) | Good | | Source CRUD/import/export/preferred | Medium | Strong (`api_sources_test.rs`) | Medium (API-driven) | Good | @@ -107,6 +116,7 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs ## 6) Detailed QA / Refactoring Plan ### Phase 1 (1-2 weeks): close highest-risk deterministic gaps + - Add scheduler integration suite: - due schedule executes once - `last_run_at` blocks double-run @@ -119,12 +129,14 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs - Add Brave Search integration path with mocked Brave API and stored encrypted key fixture. ### Phase 2 (1 week): non-functional policy tests + - Add pipeline integration tests for: - `max_age_days` filtering (`filtered_too_old` assertions) - user/provider rate-limit behavior under contention - cancellation mid-batch and partial-save invariants. ### Phase 3 (1 week): E2E realism upgrades + - Convert at least 3 API-heavy E2E scenarios to UI-driven workflows: - theme create/update/delete - source add/import/preferred/delete @@ -132,6 +144,7 @@ Scope: `docs/requirements.md`, `docs/functional_specs.md`, `docs/technical_specs - Keep API shortcuts only for setup/cleanup. ### Phase 4 (2-3 days): documentation and gate hardening + - Generate test inventory automatically (counts, pass/fail) and publish into QA docs. - Split CI lanes: - deterministic required lane (unit/integration/mock-e2e)