docs: add consolidated requirements.md and functional_specs.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 months ago · 116189d11b
parent fa793de8bf
commit 116189d11b
3 changed files with 698 additions and 0 deletions
--- a/docs/audit/test-coverage-gaps.md
+++ b/docs/audit/test-coverage-gaps.md
@ -0,0 +1,281 @@
 # Test Coverage Gap Analysis
 **Date:** 2026-03-27
 **Scope:** Backend integration tests (`backend/tests/`), backend unit tests (`backend/src/`), E2E tests (`e2e/tests/`)
 ---
 ## Summary
 | Priority | Count |
 |----------|-------|
 | Critical | 2     |
 | High     | 5     |
 | Medium   | 5     |
 | Low      | 3     |
 ---
 ## Critical Gaps
 ### GAP-01 — No integration tests for the Themes API (`/api/v1/themes`)
 **Description:**
 There is no `backend/tests/api_themes_test.rs` file. The themes handlers (`GET`, `POST`, `PUT`, `DELETE /api/v1/themes/:id`) have four endpoints with non-trivial logic: input validation via `CreateThemeRequest::validate()`, partial updates (`UpdateThemeRequest` with all-optional fields), ownership isolation (other users must get 404, not 403), and a cascade constraint (deleting a theme that still has sources assigned). None of these code paths are exercised by any integration test. The pipeline tests create a theme as setup scaffolding but do not test the themes API surface itself.
 **Priority:** Critical
 **What to add:**
 Create `backend/tests/api_themes_test.rs` covering:
 - `GET /themes` — unauthenticated returns 401; returns empty list for new user; returns created theme
 - `POST /themes` — creates theme with minimal required fields; applies defaults for `max_items_per_category`, `max_age_days`, `summary_length`; validates empty name (422); validates empty categories list (422); validates `max_items_per_category` out of range (422); validates `summary_length` out of range (422); validates >20 categories (422)
 - `PUT /themes/:id` — partial update (name only); partial update (categories only); returns 404 for non-existent id; returns 404 (not 403) when another user's theme id is used
 - `DELETE /themes/:id` — returns 204 on success; returns 404 for non-existent id; returns 404 (not 403) for another user's theme id
 - Ownership isolation — user A's themes are not visible to user B
 ---
 ### GAP-02 — No integration tests for the Article History API (`/api/v1/article-history`, `/api/v1/syntheses/:id/provenance`)
 **Description:**
 Two endpoints exist at the router level (`GET /article-history`, `DELETE /article-history`, `GET /syntheses/:id/provenance`) with no integration tests whatsoever. The pipeline test verifies that rows are written to `article_history` via a raw SQL check, but nothing tests the HTTP endpoints. The `provenance` endpoint in particular is essential for users auditing what was fetched per synthesis — a 404 from a bad join or wrong user scope would be silently untested.
 **Priority:** Critical
 **What to add:**
 Create `backend/tests/api_article_history_test.rs` covering:
 - `GET /article-history` — unauthenticated returns 401; returns empty list for new user; returns entries after a pipeline run; user isolation (user A's history not visible to user B)
 - `DELETE /article-history` — unauthenticated returns 401; clears only the requesting user's history; returns 204
 - `GET /syntheses/:id/provenance` — unauthenticated returns 401; returns correct trace entries for a known synthesis; returns 404 for non-existent synthesis id; returns 404 (not 403) for another user's synthesis
 ---
 ## High Gaps
 ### GAP-03 — No pipeline integration test for the Brave Search code path (`use_brave_search: true`)
 **Description:**
 The synthesis pipeline has two distinct code branches at line 603 of `synthesis.rs` based on `settings.use_brave_search`. All three pipeline integration tests in `pipeline_test.rs` set `"use_brave_search": false`. The entire Brave Search path — `resolve_brave_key`, `crate::services::brave_search::search`, `filter_phase2_url`, article classification via Brave results, and `filtered_diversity` traces for brave URLs — is never exercised by any integration test. The `brave_search` unit test only checks the `freshness_from_days` mapping helper.
 **Priority:** High
 **What to add:**
 Add a pipeline test in `pipeline_test.rs` (or a new `api_brave_search_pipeline_test.rs`) that:
 - Creates a user with `use_brave_search: true` and a stored `brave_search` API key
 - Injects a mock LLM that returns article URLs plus a mock HTTP server for the Brave API response
 - Asserts that articles from the Brave path appear in the synthesis sections
 - Asserts `source_type = "brave_search"` in `article_history`
 Also add to `api_keys_test.rs`:
 - `POST /user/api-keys` with `provider_name: "brave_search"` succeeds and stores the key
 - `POST /user/api-keys/brave_search/test` exercises the separate Brave key test branch (currently only LLM providers are tested via the test endpoint)
 ---
 ### GAP-04 — `assign_category` function has no unit tests
 **Description:**
 `assign_category` (line 1148 of `synthesis.rs`) is a pure function responsible for matching the LLM's output key (e.g. `"category_0"`) to the user's configured category list, extracting title and summary, and returning `None` when the key is not found. It is called on every article during both the source-scrape pass and the Gemini grounding pass. The synthesis unit test module (`mod tests`) covers `parse_llm_output`, `rotate_sources`, `normalize_article_url`, and `sanitize_error_message`, but `assign_category` is absent despite being equally pure and testable.
 **Priority:** High
 **What to add:**
 Add unit tests inside `synthesis.rs` `mod tests`:
 - Matching key in a single-category map returns correct `(key, name, title, summary)` tuple
 - Matching key when multiple categories exist picks the correct one
 - Non-matching key returns `None`
 - Item with empty title or summary is handled gracefully
 ---
 ### GAP-05 — Source diversity (`max_articles_per_source`) enforcement has no dedicated test
 **Description:**
 `max_articles_per_source` is implemented in the pipeline (the `filtered_diversity` status in `synthesis.rs` at line 403 and line 1214) and is a live setting that ships to users, but there is no integration or unit test verifying that articles from the same domain get capped. The `pipeline_test.rs` tests configure `max_articles_per_source` values but never assert that the diversity cap was actually triggered or that the trace status `"filtered_diversity"` appears.
 **Priority:** High
 **What to add:**
 In `pipeline_test.rs` add a test that:
 - Sets `max_articles_per_source: 1`
 - Injects a mock LLM that returns two URLs from the same domain (e.g. `https://blog.example.com/a` and `https://blog.example.com/b`) and a third from a different domain
 - Asserts only one article from `blog.example.com` appears in the final sections
 - Asserts that an `article_history` row with `status = "filtered_diversity"` exists for the dropped URL
 ---
 ### GAP-06 — No E2E spec for the Themes management UI
 **Description:**
 The `e2e/tests/` directory contains dedicated specs for sources, settings, admin providers, and generation. There is no `themes.spec.ts`. The `generation-live.spec.ts` creates a theme via direct API call as setup scaffolding but never tests the Themes page UI: creating a theme through the form, editing it, deleting it, or validating that validation errors surface correctly. This is particularly risky because themes are the primary user configuration surface for the synthesis pipeline.
 **Priority:** High
 **What to add:**
 Create `e2e/tests/themes.spec.ts` covering:
 - Create a theme via the UI form and verify it appears in the list
 - Edit a theme name and save; verify the updated name persists after page reload
 - Delete a theme; verify it disappears from the list
 - Submit the form with an empty name; verify a validation error is shown
 ---
 ### GAP-07 — Article history deduplication behavior has no integration test
 **Description:**
 The history deduplication mechanism (checking `url_hash` in `article_history` before including a URL) is a core pipeline behavior to prevent the same article appearing in multiple consecutive weekly syntheses. While `pipeline_test.rs` verifies rows are written, no test runs the pipeline twice and asserts that the second run skips URLs already seen (producing `article_history` rows with `status = "filtered_already_seen"` or similar). The `article_history_days: 0` setting (which disables deduplication entirely) is also untested in the pipeline.
 **Priority:** High
 **What to add:**
 In `pipeline_test.rs` add a test that:
 - Runs the mock pipeline once, recording which URLs were `"used"`
 - Runs the pipeline a second time with the same source URLs
 - Asserts the second run does not re-include URLs already present in `article_history` for that user
 - Optionally: a variant with `article_history_days: 0` confirming re-inclusion is allowed
 ---
 ## Medium Gaps
 ### GAP-08 — No unit tests for `CreateThemeRequest::validate()`
 **Description:**
 The `validate()` method on `CreateThemeRequest` contains seven distinct validation rules (empty name, name >200 chars, empty theme, empty categories, >20 categories, empty category element, range checks on `max_items_per_category`/`max_age_days`/`summary_length`). None of these rules are tested by unit tests in `backend/src/models/theme.rs`, which has no `#[cfg(test)]` block at all. The integration tests that will be added in GAP-01 will exercise this indirectly via HTTP, but pure unit tests would be faster and more granular.
 **Priority:** Medium
 **What to add:**
 Add `#[cfg(test)] mod tests` in `backend/src/models/theme.rs` with unit tests for each validation rule, mirroring the style of `backend/src/models/settings.rs`'s existing `mod tests`.
 ---
 ### GAP-09 — `api_keys_test.rs` only tests `gemini` and `openai` provider keys
 **Description:**
 The `api_keys_test.rs` tests use `provider_name: "gemini"` for almost every test and `"openai"` in one test. The `"anthropic"` provider and the special-cased `"brave_search"` provider are never used. The `brave_search` provider has distinct handler logic (lines 124–144 of `api_keys.rs`) that routes to `brave_search::test_api_key` instead of the LLM factory — this branch is entirely untested.
 **Priority:** Medium
 **What to add:**
 Add to `api_keys_test.rs`:
 - Store and retrieve an `anthropic` API key
 - Store a `brave_search` API key and call `POST /user/api-keys/brave_search/test` — verify the response shape is `{ success: bool, message: string }`
 - Verify `POST /user/api-keys/invalid_provider/test` returns 404 when no key exists
 ---
 ### GAP-10 — No tests for the `source_scraper` service against real network-style responses
 **Description:**
 `source_scraper.rs` has unit tests that cover `extract_links_from_html` and related HTML parsing. However, there are no tests for the `scrape_source` function path that sends real HTTP requests to extract article links, respects `max_links_per_source`, or handles redirect chains. These are exercised only incidentally by E2E tests that hit real URLs. A wiremock or similar in-process HTTP server could isolate these tests from external dependencies.
 **Priority:** Medium
 **What to add:**
 Add integration-style tests in `source_scraper.rs` or a new test file using `wiremock` or `axum`'s test utilities to stand up a local HTTP server, covering:
 - Fetch and parse links from a mock HTML page respecting `max_links_per_source`
 - SSRF blocked URL returns `AppError::BadRequest`
 - Redirect to a private IP is blocked by the SSRF middleware
 ---
 ### GAP-11 — No E2E test for the Article History / Provenance UI
 **Description:**
 No E2E spec exercises the article history list page or the per-synthesis provenance view. These are both user-facing features for auditing synthesis quality. Their absence from E2E means a broken route or rendering error could go undetected before release.
 **Priority:** Medium
 **What to add:**
 Add cases to a new or existing E2E spec:
 - Navigate to the article history page; verify it loads without error
 - After triggering a synthesis (can reuse `generation-live.spec.ts` setup), open the synthesis detail and verify the provenance section renders at least one entry
 ---
 ### GAP-12 — `export.rs` unit tests do not cover PDF export shape
 **Description:**
 `backend/src/services/export.rs` has a `#[cfg(test)]` module. The integration tests in `api_export_test.rs` cover HTTP status codes for Markdown and PDF endpoints (auth, not-found, valid). However, neither the unit tests nor the integration tests assert the actual structure of the PDF output (e.g. that the PDF bytes start with `%PDF-`). A regression that produces zero-length or malformed PDF bytes would not be caught.
 **Priority:** Medium
 **What to add:**
 In `api_export_test.rs` add a test that:
 - Requests `GET /syntheses/:id/export/pdf` for a valid synthesis
 - Reads the response body bytes
 - Asserts the first 4 bytes are `%PDF` (the PDF magic number)
 ---
 ## Low Gaps
 ### GAP-13 — No test for `session` expiry / stale cookie rejection
 **Description:**
 `api_auth_test.rs` covers logout and valid session use, but no test deletes a session row directly from the database and then confirms the next authenticated request returns 401. The session validation path in the auth middleware (`middleware/auth.rs`) is only tested via unit tests for token extraction, not via DB-backed expiry.
 **Priority:** Low
 **What to add:**
 Add an integration test that:
 - Creates an authenticated user and obtains a session token
 - Deletes the session row directly via `sqlx::query`
 - Asserts the next `GET /api/v1/settings` with the old session token returns 401
 ---
 ### GAP-14 — Generation endpoint does not have a test for missing LLM API key
 **Description:**
 If a user triggers generation but has no API key stored for their selected provider, the pipeline will fail during `resolve_provider_and_key`. No test covers this specific error path. The 202 response is returned before the async pipeline fails, so the SSE `error` event is the user-visible signal — but nothing tests that it emits the right error type.
 **Priority:** Low
 **What to add:**
 Add a `pipeline_test.rs` test that:
 - Creates a user with no API keys stored
 - Runs the pipeline directly (not via HTTP) with a mock that requires a key
 - Asserts the pipeline returns an error with a sanitized message (not leaking internal details)
 ---
 ### GAP-15 — `prompts.rs` unit tests do not cover `use_brave_search` or `search_agent_behavior` prompt injection
 **Description:**
 `backend/src/services/prompts.rs` has a `#[cfg(test)]` module. The existing tests check basic prompt construction. However, the `search_agent_behavior` field (injected as a custom instruction into the prompt) and the Brave Search-specific prompt variant are not tested. A whitespace error or missing format string in the `search_agent_behavior` injection would go undetected.
 **Priority:** Low
 **What to add:**
 Add unit tests to `prompts.rs` `mod tests`:
 - When `search_agent_behavior` is non-empty, the returned prompt string contains the custom instruction verbatim
 - When `search_agent_behavior` is empty, the prompt does not contain a spurious empty line or placeholder
 ---
 ## Coverage Matrix
 | Area | Integration Test | Unit Test | E2E Test |
 |------|-----------------|-----------|----------|
 | Auth (register, login, verify, logout) | Yes | Partial | No |
 | Settings CRUD | Yes | Yes | Yes |
 | Sources CRUD + CSV | Yes | Yes | Yes |
 | **Themes CRUD** | **NO** | **NO** | **NO** |
 | API Keys CRUD + test | Partial (gemini/openai only) | Yes | No |
 | Syntheses CRUD + pagination | Yes | Yes | No |
 | Generation trigger + SSE | Yes (202 + conflict) | Yes (JobStore) | Yes |
 | Pipeline (mock LLM, source scrape path) | Yes (3 tests) | Yes | No |
 | **Pipeline (Brave Search path)** | **NO** | Partial | No |
 | **Pipeline (max_articles_per_source cap)** | **NO** | No | No |
 | **Pipeline (article history dedup)** | **NO** | No | No |
 | Admin providers + rate limits | Yes | Yes | Yes |
 | Admin user management + roles | Yes | Yes | No |
 | Export (Markdown, PDF, email) | Yes | Yes | Yes (Markdown only) |
 | **Article History endpoints** | **NO** | No | **NO** |
 | **Synthesis Provenance endpoint** | **NO** | No | **NO** |
 | CSRF middleware | Yes | Yes | No |
 | SSRF prevention | No | Yes | No |
 | Encryption at rest | Yes | Yes | No |
 | Brave Search key test endpoint | **NO** | No | No |
--- a/docs/functional_specs.md
+++ b/docs/functional_specs.md
@ -0,0 +1,276 @@
 # AI Weekly Synth -- Functional Specification
 ## 1. User Journeys
 ### 1.1 Registration
 1. User navigates to the registration page and enters their email and optional display name.
 2. A Cloudflare Turnstile captcha is completed.
 3. The system sends a magic link email.
 4. User clicks the link to verify their account and is logged in automatically.
 5. A 30-day session cookie is set. The user is redirected to the home page.
 ### 1.2 Login
 1. User enters their email on the login page.
 2. A Turnstile captcha is completed.
 3. A magic link is sent. The user clicks it and is authenticated.
 4. If the link expires or is invalid, the user is prompted to request a new one.
 ### 1.3 Configure a Theme
 1. User navigates to "Personnaliser les syntheses" (theme management page).
 2. User selects an existing theme from the dropdown or clicks "Creer un nouveau theme".
 3. The theme form shows:
   - **Name**: a display label for the theme.
   - **Search topic**: the subject the AI uses to search for news (e.g. "Intelligence Artificielle").
   - **Categories**: an ordered list of user-defined category names. Categories can be added and removed. The system always adds an implicit "Autre" overflow category.
   - **Max age (days)**: how old articles can be.
   - **Max items per category**: cap per category.
   - **Summary length**: slider with three positions -- Court (3-4 lines), Moyen (6-8 lines), Detaille (12-15 lines).
 4. User saves the theme.
 ### 1.4 Add Personalized Sources
 1. On the theme management page, below theme settings, the sources section shows sources scoped to the selected theme.
 2. User adds sources individually (title + URL) or via:
   - **CSV import**: upload a `.csv` file with `Titre,URL` columns. Auto-detects comma/semicolon delimiters, skips header rows, prepends `https://` to bare URLs.
   - **Bulk text import**: paste multiple sources in `Nom;URL` format, one per line.
   - **CSV export**: download all sources for the theme as a CSV file.
 3. Sources can be marked as **preferred** (prioritaire) via checkboxes. Preferred sources are processed first during generation. A counter shows how many sources are preferred.
 4. Sources can be deleted individually.
 ### 1.5 Generate a Synthesis
 1. User navigates to "Nouvelle Synthese" and selects a theme from the dropdown.
 2. The page shows the active provider and model.
 3. User clicks "Lancer la generation".
 4. Progress is streamed in real-time via SSE. The page shows the current step:
   - "Sources personnalisees" (Phase 1)
   - "Recherche web" (Phase 2)
   - "Sauvegarde" (final step)
 5. User can leave the page; generation continues in the background.
 6. User can stop the generation early; articles collected so far are saved.
 7. On completion, user is redirected to the synthesis detail page.
 ### 1.6 View a Synthesis
 1. The home page lists all syntheses as cards, showing the week number, theme name badge, a preview of articles, and article count.
 2. Syntheses can be sorted by date or by theme.
 3. Clicking a card opens the detail page.
 4. The detail page displays sections (categories) with article titles, summaries, and links. Two display modes are available: compact and full.
 5. From the detail page, the user can:
   - View article provenance (history of candidate articles processed during generation).
   - View LLM call logs (every AI call made during generation, with prompts, responses, and timing).
 ### 1.7 Export a Synthesis
 From the synthesis detail page:
 - **Email**: enter a recipient address or click "S'envoyer a soi-meme". The synthesis is sent as a formatted email via Resend.
 - **Markdown**: download as a `.md` file.
 - **PDF**: download as a `.pdf` file.
 ## 2. Feature Details
 ### 2.1 Multi-Theme
 Each user can create multiple themes. A theme groups together:
 - Content settings (search topic, categories, max items, max age, summary length)
 - Personalized sources
 - Generated syntheses
 Themes are fully independent. Deleting a theme preserves its existing syntheses (displayed with a "Theme supprime" badge).
 The generate page requires selecting a theme before launching. The home page shows a theme badge on each synthesis card and supports sorting by theme.
 ### 2.2 Categories
 Categories are user-defined per theme. Users add and remove category names in the theme editor. The system always appends an implicit "Autre" category to catch articles that do not match any user-defined category, or articles from categories that have reached their max items cap.
 If no categories are configured, the only available category is "Autre".
 ### 2.3 Preferred Sources
 Sources can be marked as preferred. During generation, preferred sources are extracted and processed before non-preferred sources. Within each extraction wave, URLs from preferred sources are also shuffled and placed before other URLs. This maximizes the chance that articles from preferred sources fill the synthesis.
 ### 2.4 Scheduled Generation
 Each theme can have an optional schedule with:
 - **Enabled/disabled toggle**
 - **Days**: selection of days of the week (Mon-Sun)
 - **Time**: execution time in UTC (HH:MM)
 - **Email recipients**: up to 3 email addresses
 When a schedule fires, the system generates the synthesis and emails it to all listed recipients. Schedules are checked every 60 seconds. A `last_run_at` timestamp prevents double-runs on the same day. Jobs run sequentially to avoid overwhelming LLM rate limits.
 Changes to the schedule are saved immediately (auto-save).
 ### 2.5 Brave Search
 An optional alternative to LLM-powered web search in Phase 2. When enabled:
 - The user provides a Brave Search API key (stored encrypted alongside LLM keys).
 - Phase 2 queries the Brave Search API with the theme topic, filtered by article freshness.
 - Results are scraped and classified/summarized by the LLM, following the same pipeline as Phase 1.
 When the Brave key is deleted, the toggle is automatically disabled. If the toggle is on but no key is present at generation time, the system returns an error.
 ## 3. Generation Pipeline
 ### 3.1 Overview
 Generation follows a two-phase pipeline. Phase 1 processes the user's personalized sources. Phase 2 fills remaining category gaps via web search. Both phases produce articles classified into user-defined categories with titles, summaries, and source URLs.
 ### 3.2 Initialization
 Before generation starts:
 1. Load theme settings (categories, search topic, max items, max age, summary length) and global user settings (provider, models, batch size, rate limits, etc.).
 2. Decrypt the user's LLM API key and create the provider instance.
 3. Clean up old article history and LLM call logs.
 4. Load personalized sources for the selected theme.
 5. Initialize tracking: per-category article counts, per-domain source counts, seen URLs set, article history hashes.
 ### 3.3 Phase 1: Personalized Sources
 Skipped if the user has no sources for the theme.
 **Step 1 -- Windowed source extraction:**
 Sources are split into waves of `source_extraction_window` size (default 3). Sources are rotated so extraction starts after the last source used in a previous generation (rolling window). Preferred sources are placed before non-preferred sources within the rotation order.
 For each wave:
 1. Extract article links from all sources in the wave in parallel (bounded concurrency of 5). Link extraction uses either LLM analysis of the page content or HTML `<a>` tag parsing (configurable).
 2. Deduplicate candidate URLs and filter against article history (previously seen articles are skipped).
 3. Shuffle remaining candidates, with URLs from preferred sources placed first.
 4. Process articles in batches of `batch_size`:
   - **Scrape**: fetch article pages in parallel. Validate content (reject empty pages, soft-404s, pages that are too old). Extract original title from `og:title`, `<h1>`, or `<title>`.
   - **Classify/summarize**: send article content to the LLM. The LLM assigns a category and generates a title and summary. Summary length varies based on the `summary_length` setting (more detail = more article body sent to the LLM).
 5. Check if the synthesis is full (total articles across all categories reaches the cap). If full, skip remaining waves.
 **Source diversity**: a per-domain cap (`max_articles_per_source`) prevents any single source from dominating.
 ### 3.4 Phase 2: Web Search Fallback
 Skipped if all user-defined categories are already filled.
 The system computes category gaps (how many articles each category still needs), then follows one of two paths:
 **Path A -- Brave Search** (when `use_brave_search` is enabled):
 1. Query the Brave Search API with the theme topic and freshness filter.
 2. Filter results: reject homepage URLs, deduplicate against Phase 1, check article history, apply source diversity cap.
 3. Scrape and classify/summarize results using the same batched pipeline as Phase 1.
 **Path B -- LLM Web Search** (default):
 1. Send a search prompt to the LLM with the theme, categories, and gap counts. The LLM uses web grounding to find articles and returns structured results.
 2. Filter results using the same filters as Path A.
 3. Scrape each result to validate it. Keep the LLM-provided title and summary (no re-classification).
 ### 3.5 Finalization
 1. If no articles were collected across both phases, return an error.
 2. Order sections: user-defined categories first (in their configured order), then "Autre" if non-empty.
 3. Save the synthesis to the database with status "completed".
 4. Record all used articles in article history for future deduplication.
 ## 4. Settings Overview
 ### 4.1 Per-Theme Settings
 Managed on the theme management page. Each theme has its own values.
 | Setting | Description | Default |
 |---------|-------------|---------|
 | Name | Display label for the theme | -- |
 | Search topic | Subject for AI search queries | -- |
 | Categories | Ordered list of category names | [] |
 | Max age (days) | Article recency filter | 7 |
 | Max items per category | Cap per category | 4 |
 | Summary length | Detail level: 1=Court, 2=Moyen, 3=Detaille | 3 |
 ### 4.2 Global User Settings
 Managed on the settings page. Apply across all themes.
 | Setting | Description | Default |
 |---------|-------------|---------|
 | Provider | LLM provider (Gemini, OpenAI, Anthropic) | -- |
 | Research model | Model for scraping/classification | Admin default |
 | Web search model | Model for web search | Admin default |
 | Search agent behavior | Custom instructions for AI research | Default prompt |
 | Use Brave Search | Enable Brave Search for Phase 2 | false |
 | Batch size | Articles processed in parallel | 5 |
 | Source extraction window | Sources per extraction wave | 3 |
 | Max articles per source | Per-domain diversity cap | -- |
 | Max links per source | Links extracted per source page | 15 |
 | Rate limit (max requests) | LLM call throttling | Admin default |
 | Rate limit (time window) | Throttling window | Admin default |
 | Article history (days) | History retention period | -- |
 ### 4.3 Settings Import/Export
 Users can export their global settings as a JSON file and import settings from a previously exported file. Import merges uploaded values over defaults; missing fields fall back to default values. API keys can optionally be included in the export (with a warning that they will be in plaintext).
 ## 5. Admin Features
 ### 5.1 Provider Management
 Admins configure which LLM providers and models are available to users:
 - Add providers with a unique identifier and display name.
 - For each provider, configure two model lists: scraping/extraction models and web search models.
 - Set a default model for each category.
 - Enable or disable providers.
 - Delete providers entirely.
 Users select from the admin-curated list. If a user's selected provider is removed, they see a warning to select another.
 ### 5.2 Rate Limit Configuration
 Admins set default rate limits per provider (max requests / time window in seconds). These defaults apply to users who have not overridden the values in their own settings.
 ### 5.3 User Management
 Admins can:
 - View all registered users (email, name, role, registration date).
 - Promote a user to admin or demote an admin to user.
 - Admins cannot modify their own role.
 ## 6. Export and Sharing
 ### 6.1 Email
 From the synthesis detail page, users enter a recipient email address or click "S'envoyer a soi-meme" to use their own address. The synthesis is sent as a formatted HTML email via the Resend API.
 Scheduled generation also sends emails automatically to up to 3 configured addresses per theme.
 ### 6.2 PDF
 A PDF export is available from the synthesis detail page. The PDF contains all sections with article titles, summaries, and source URLs.
 ### 6.3 Markdown
 A Markdown export is available from the synthesis detail page. The file can be saved or pasted into other tools.
 ## 7. Article History and Observability
 ### 7.1 Article History
 Every article encountered during generation is recorded in the article history with its status:
 - **used**: included in the final synthesis.
 - **filtered_history**: skipped because it was seen in a previous generation.
 - **filtered_diversity**: skipped due to per-domain cap.
 - **filtered_empty**: scrape returned no content or a soft-404.
 - **filtered_too_old**: article older than the max age setting.
 - **filtered_homepage**: URL was a homepage, not a specific article.
 - **filtered_cross_phase_dedup**: URL already seen in a previous phase.
 Users can view the article history per synthesis (provenance view) or globally. History can be cleared entirely.
 ### 7.2 LLM Call Logs
 Every LLM call during generation is logged with:
 - Call type (link extraction, classify/summarize, web search)
 - Model used
 - System prompt and user prompt
 - Response
 - Duration
 - Associated article URL (for classify calls)
 Logs are viewable per synthesis from the detail page.
--- a/docs/requirements.md
+++ b/docs/requirements.md
@ -0,0 +1,141 @@
 # AI Weekly Synth -- Requirements
 ## 1. Product Vision
 AI Weekly Synth is a self-hosted web application that generates AI-powered weekly news syntheses. Users define topics of interest (themes), add personalized sources, and let the application search the web, validate articles, and produce structured summaries organized by category.
 The application is designed for individuals or small teams who want an automated, curated news digest without relying on third-party newsletter services.
 ## 2. Target Users
 - **End users**: professionals or enthusiasts who follow one or more topics (e.g. AI, cybersecurity, finance) and want a weekly summary delivered by email or available on-demand.
 - **Administrators**: the instance operator who manages available LLM providers, rate limits, and user accounts.
 ## 3. Core Features
 ### 3.1 Multi-Theme Support
 - Users create multiple themes, each with its own search topic, categories, and content settings.
 - Each theme has its own set of personalized sources.
 - Syntheses are generated per theme and tagged accordingly.
 - Themes can be created, edited, and deleted independently. Deleting a theme preserves its existing syntheses.
 ### 3.2 Synthesis Generation
 - On-demand generation triggered by the user for a selected theme.
 - Two-phase pipeline:
  - **Phase 1 (Personalized Sources)**: extracts article links from user-configured sources, scrapes content, classifies and summarizes each article into the theme's categories.
  - **Phase 2 (Web Search Fallback)**: fills remaining category gaps using either Brave Search API or LLM-powered web search.
 - Real-time progress streaming via SSE so the user can monitor generation status.
 - Generation is capped at 15 minutes with automatic timeout.
 ### 3.3 Scheduled Generation
 - Users configure a per-theme schedule: selected days of the week, time (UTC), and up to 3 email recipients.
 - The application runs scheduled jobs automatically in the background, generating the synthesis and emailing it to all configured recipients.
 - No external cron required; the scheduler is an internal background task.
 ### 3.4 Personalized Sources
 - Users add web sources (blogs, news sites) per theme.
 - Sources can be imported in bulk via text input, CSV upload, or added individually.
 - Sources can be exported as CSV.
 - Sources can be marked as **preferred** (prioritized during generation -- processed before non-preferred sources).
 ### 3.5 Brave Search Integration
 - Optional alternative to LLM web search for Phase 2.
 - Users provide their own Brave Search API key.
 - When enabled, Phase 2 queries the Brave Search API instead of using LLM web grounding, then scrapes and classifies the results.
 ### 3.6 Export and Sharing
 - **Email**: send a synthesis to any email address (or to self) via Resend.
 - **PDF**: download a synthesis as a PDF file.
 - **Markdown**: download a synthesis as a Markdown file.
 ### 3.7 Settings
 #### Per-theme settings (content)
 - Theme name and search topic
 - Categories (user-defined list)
 - Max age of articles (days)
 - Max items per category
 - Summary detail level (short / medium / detailed)
 #### Global settings (pipeline and AI)
 - LLM provider and model selection (research model + web search model)
 - Search agent behavior (custom instructions for the AI research prompt)
 - Brave Search toggle and API key
 - Batch size (articles processed in parallel)
 - Source extraction window (number of sources per extraction wave)
 - Max articles per source (diversity cap)
 - Max links extracted per source
 - Rate limiting (max requests / time window)
 - Article history retention (days)
 - Settings import/export (JSON)
 ### 3.8 Authentication
 - Passwordless authentication via magic link emails.
 - Cloudflare Turnstile captcha on login and registration.
 - 30-day session cookies (HttpOnly, SameSite).
 ## 4. User Roles
 ### 4.1 User (default)
 - Register and log in via magic link.
 - Create and manage themes (CRUD).
 - Add and manage personalized sources per theme.
 - Configure generation settings and API keys.
 - Generate syntheses on demand or via schedule.
 - View, delete, and export syntheses.
 - View article history and LLM call logs per synthesis.
 ### 4.2 Admin
 All user capabilities, plus:
 - **Provider management**: add, edit, enable/disable, and remove LLM providers and their available models. Users select from admin-curated providers.
 - **Rate limit configuration**: set default rate limits per provider (max requests / time window). Users can override with their own values.
 - **User management**: view all users, promote users to admin or demote admins to user.
 The first admin is created via a CLI command (`create-admin`).
 ## 5. Non-Functional Requirements
 ### 5.1 Security
 - API keys (LLM, Brave Search) encrypted at rest with AES-256-GCM using a master encryption key.
 - SSRF prevention in the scraper (rejects private/loopback IPs).
 - CSRF protection via `X-Requested-With` header validation.
 - Session-based authentication with HttpOnly/SameSite cookies.
 ### 5.2 Performance
 - Configurable rate limiting for LLM API calls (per-user override or admin default).
 - Batched parallel scraping and classification to maximize throughput.
 - Windowed source extraction to avoid unnecessary work when the synthesis fills early.
 - Source diversity cap to prevent a single domain from dominating results.
 - Article history deduplication to avoid re-processing previously seen articles.
 - 15-minute generation timeout.
 ### 5.3 Self-Hosted
 - Single Docker Compose deployment (application + PostgreSQL).
 - No external dependencies beyond user-provided API keys and the Resend email service.
 - Single-tenant: one instance per deployment.
 - Users bring their own LLM API keys (no shared API key).
 ### 5.4 Internationalization
 - i18n-ready architecture (all UI strings externalized).
 - French is the only language currently supported.
 ### 5.5 Reliability
 - Hourly session cleanup background task.
 - Job store with TTL for expired generation jobs.
 - Scheduled generation with double-run prevention (`last_run_at` tracking).
 - Panic recovery and timeout handling for generation tasks.