diff --git a/docs/superpowers/specs/2026-03-24-article-history-design.md b/docs/superpowers/specs/2026-03-24-article-history-design.md new file mode 100644 index 0000000..f3dc814 --- /dev/null +++ b/docs/superpowers/specs/2026-03-24-article-history-design.md @@ -0,0 +1,131 @@ +# Design: Article History — Prevent Duplicate Articles Across Syntheses + +**Date**: 2026-03-24 +**Scope**: Persistent article URL history to prevent the same article from appearing in multiple syntheses + +--- + +## Context + +Article URLs are only deduplicated within a single synthesis (via `dedup_by_url` and `seen_urls`). Across syntheses, there is no dedup — if an article stays on a blog's front page, it reappears every week. The `source_diversity_window` only avoids domains, not specific URLs. + +## Approach + +New `article_history` table stores all article URLs per user. During generation, candidate articles are filtered against this history. A configurable TTL allows articles to reappear after N days. + +## New Table: `article_history` + +```sql +CREATE TABLE article_history ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE, + url_hash TEXT NOT NULL, + url TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE UNIQUE INDEX idx_article_history_user_url ON article_history(user_id, url_hash); +``` + +- `url_hash`: SHA-256 of lowercased URL (fast indexed lookup, avoids indexing long text) +- Scoped per `user_id` — each user has independent history +- `UNIQUE(user_id, url_hash)` constraint — inserts use `ON CONFLICT DO NOTHING` to silently skip duplicates + +## New User Setting + +- **Field:** `article_history_days: i32` (default 90, range 0-365, 0 = disabled) +- **Label:** "Historique des articles (jours)" +- **Migration:** `ALTER TABLE settings ADD COLUMN article_history_days INTEGER NOT NULL DEFAULT 90` + +## Pipeline Integration + +**Where history filtering happens:** After scraping + filtering empty content, before classification. Avoids wasting LLM classification calls on articles that will be filtered out. + +### Phase 1 (personalized sources): + +1. Scrape source pages → extract article links +2. Scrape candidate articles → filter empty content +3. **Filter against article_history** — query `check_urls_exist(user_id, url_hashes)`, remove matches +4. **If under-filled: 1 retry** — go back to step 1 with the same sources, excluding already-fetched URLs (both history-filtered and previously-scraped URLs in this run). Only 1 retry attempt. +5. LLM classification → fill categories + +### Phase 2 (web search fallback): + +1. LLM search → parse + filter homepage + dedup +2. **Filter against article_history** — remove already-seen URLs BEFORE scraping (saves HTTP requests) +3. Scrape remaining articles → filter empty +4. LLM classification → fill remaining + +### After saving synthesis: + +Insert all article URLs from the final synthesis into `article_history`: +- `insert_urls(user_id, urls)` — batch insert with SHA-256 hashes + +### Cleanup: + +Before each generation, delete entries older than `article_history_days` for this user: +- `cleanup_old(user_id, days)` — `DELETE FROM article_history WHERE user_id = $1 AND created_at < now() - interval '$2 days'` +- When `article_history_days = 0`, skip both filtering and insertion (feature disabled) + +## DB Module: `db/article_history.rs` + +Three functions: + +```rust +/// Check which URLs already exist in history. Returns the set of url_hashes that exist. +pub async fn check_urls_exist(pool: &PgPool, user_id: Uuid, url_hashes: &[String]) -> Result, AppError> + +/// Insert article URLs into history (batch). +pub async fn insert_urls(pool: &PgPool, user_id: Uuid, urls: &[(String, String)]) -> Result<(), AppError> +// urls is Vec<(url, url_hash)> + +/// Delete history entries older than N days. +pub async fn cleanup_old(pool: &PgPool, user_id: Uuid, days: i32) -> Result +``` + +**URL normalization before hashing:** URLs must be normalized before hashing to catch common variations: +1. Lowercase the URL +2. Strip fragment (`#...`) +3. Strip trailing slash from path (unless path is just `/`) +4. Strip known tracking query parameters: `utm_source`, `utm_medium`, `utm_campaign`, `utm_term`, `utm_content`, `ref`, `source`, `fbclid`, `gclid` +5. If no query parameters remain after stripping, remove the `?` entirely + +Add a `normalize_article_url(url: &str) -> String` utility (in `synthesis.rs` or a shared util). Use `url::Url::parse` for reliable parsing. + +**URL hashing:** `sha256(normalize_article_url(url))` — reuse existing `crate::util::token::hash_token` (which does SHA-256 → hex). + +## Retry Logic (Phase 1) + +When articles are filtered out by history and categories are under-filled: + +1. Collect URLs already fetched in this run (both valid and history-filtered) +2. Re-scrape source pages, excluding already-fetched URLs +3. Scrape new candidates → filter empty → filter history +4. Merge with existing results +5. Proceed to classification + +Only 1 retry attempt. "Under-filled" means the total number of valid scraped articles (after history filtering) is less than `categories.len() * max_items_per_category`. If still under-filled after retry, Phase 2 fills the gaps. + +## Files to Modify + +- **Create:** migration `20260324000015_add_article_history.sql` — table + setting column +- **Create:** `backend/src/db/article_history.rs` — check_urls_exist, insert_urls, cleanup_old +- **Modify:** `backend/src/db/mod.rs` — register module +- **Modify:** `backend/src/models/settings.rs` — add `article_history_days` field +- **Modify:** `backend/src/db/settings.rs` — add to queries +- **Modify:** `backend/src/services/synthesis.rs` — history filtering, retry logic, insert after save, cleanup before generation +- **Modify:** `frontend/src/types.ts` — add setting field +- **Modify:** `frontend/src/i18n/fr.ts` — add label +- **Modify:** `frontend/src/pages/Settings.tsx` — add number input +- **Modify:** `CLAUDE.md` — migration count to 15 +- **Modify:** `e2e/tests/generation-live.spec.ts` — update settings payload +- **Modify:** `backend/tests/api_syntheses_test.rs` — update settings payload +- **Add:** unit tests in `synthesis.rs` — history filtering logic + +## What Does NOT Change + +- `dedup_by_url` — still handles within-synthesis dedup (fast, in-memory) +- `seen_urls` HashSet — still handles cross-phase dedup within one generation +- `source_diversity_window` — still avoids domains across syntheses (complementary) +- Frontend synthesis display — no changes +- Scraper — no changes +- LLM providers — no changes