`, `og:title`), date (meta tags, JSON-LD, `<time>`), body (strip scripts/nav), soft-404 detection - Capture final URL after redirects (canonical URL) ### 1c. Filter empty content - Remove articles where scraped body text is empty (scrape failure, soft 404, too old) - Trace dropped articles in `article_history` with `status: filtered_empty` ### 1d. Filter against article history - Hash each URL (normalized: lowercase, strip fragments/UTM params/trailing slashes) - Query `article_history` for existing hashes → remove matches - Trace dropped articles with `status: filtered_history` ### 1e. Retry if under-filled - If valid articles < `categories × max_items_per_category` and history is enabled - Re-scrape source pages for NEW links (exclude already-fetched URLs) - Scrape + filter empty + filter history on retry candidates - Merge with existing valid articles - Only 1 retry attempt ### 1f. LLM classification - Send articles (title + first 500 chars of body) + categories + "Autre" to LLM - LLM returns `{assignments: [{index, category}]}` mapping each article to a category - Overflow: articles that exceed both target category AND "Autre" limits → collected in `all_overflow` - **LLM call logged** with full prompt/response/timing ### 1g. Enforce source diversity - Count domains across all categories - Remove articles where domain exceeds `max_articles_per_source` - Trace dropped articles with `status: filtered_diversity` - Recount category fill levels --- ## Phase 2: Web Search Fallback **Skipped if all user-defined categories are already filled to `max_items_per_category`.** ### 2a. Compute category gaps - For each user category: `needed = max - already_filled` - Only proceed if any category needs more ### 2b. Load recent domains for diversity - If `source_diversity_window > 0`: extract domains from last N syntheses' JSONB sections - Used as soft "avoid if possible" instruction in search prompt ### 2c. LLM web search pass - Build search prompt with theme, categories, gap counts ("find N articles for AI News, M for Cybersecurity"), recent domains to avoid, personalized source URLs - Call `provider.generate_search_pass()` with web search tool enabled - **LLM call logged** with full prompt/response/timing - Returns structured JSON: `{category_0: [{title, url, summary}], category_1: [...]}` ### 2d. Filter pipeline on search results - **Parse** LLM output into `(category_key, Vec<NewsItem>)` - **Filter homepage URLs** — drop articles with path `/` or empty - **Cross-phase dedup** — drop URLs already seen in Phase 1 - **Dedup by URL** — drop duplicate URLs within Phase 2 (case-insensitive) - **Limit articles per source** — enforce `max_articles_per_source` per domain (spread across categories first, then fill) - **Filter against article history** — BEFORE scraping (saves HTTP requests), drop already-seen URLs - Each drop traced in `article_history` with appropriate status ### 2e. Scrape web search results - Same scraping as Phase 1 (bounded concurrency, SSRF check, optional LLM extraction) - Filter empty content (scrape failures, soft 404, too old) - Trace drops ### 2f. LLM classification - Same as Phase 1 classification but with Phase 2 articles - `filled_counts` carries over from Phase 1 — categories already partially filled - Overflow collected - **LLM call logged** - Merge results into `all_scraped` --- ## "Autre" Fill-Up - Count total articles across all categories - Target = `75% × (categories × max_items_per_category)` (user categories only, "Autre" excluded from denominator) - If shortfall > 0 and overflow exists: - For each overflow article: check if domain is under `max_articles_per_source` limit - Add to `all_scraped["category_autre"]` up to the shortfall --- ## Combined Rewrite Pass - **Fail if no articles** — return error if all categories are empty - **Build rewrite prompt** — serialize all scraped articles with body content, instruct LLM to rewrite title + summary (4-5 lines) faithfully based on scraped content - **Build rewrite schema** — `minItems`/`maxItems` set to ACTUAL count per category (not user max), empty categories omitted, "Autre" included if non-empty - **LLM rewrite pass** — call `provider.generate_rewrite_pass()` with writing model - **LLM call logged** with full prompt/response/timing - **Build final sections** — map `category_N` keys to user category names, add "Autre" section if present, omit empty categories - **Restore scraped URLs** — replace any hallucinated URLs from LLM rewrite with the validated scraped URLs (matched by category + position) --- ## Save + Record - **Sanitize** — strip `\u0000` null bytes from JSON (PostgreSQL rejects them in JSONB) - **Save synthesis** — insert into `syntheses` table with `job_id`, `week` (ISO week), `sections` (JSONB), `status: completed` - **Record used articles** — insert each article URL into `article_history` with `status: used`, `synthesis_id`, `job_id`, and category name (for future dedup + provenance) --- ## Summary of LLM Calls (up to 4 per generation) | # | Call | When | Model | |---|---|---|---| | 1 | Classification Phase 1 | After Phase 1 scraping | research | | 2 | Web Search | Phase 2 start | research | | 3 | Classification Phase 2 | After Phase 2 scraping | research | | 4 | Rewrite | After both phases | writing | Plus optionally per-article calls for LLM link extraction and LLM article extraction (when those settings are enabled). ## Summary of Filtering Steps | Step | Phase | What's dropped | |---|---|---| | Empty content | 1 & 2 | Scrape failures, soft 404s, too old | | Article history | 1 & 2 | Already used in previous syntheses | | Homepage URLs | 2 | Path is `/` or empty | | Cross-phase dedup | 2 | URLs already found in Phase 1 | | URL dedup | 2 | Duplicate URLs within Phase 2 | | Source diversity | 1 & 2 | Domain exceeds `max_articles_per_source` | | Category overflow | 1 & 2

# Synthesis Generation Pipeline — Full Algorithm ## Initialization 1. **Load user settings** from DB (categories, provider, models, max_items, etc.) 2. **Cleanup** — delete old article history entries (>N days, dropped only) + truncate old LLM call logs 3. **Validate** — fail if no categories configured 4. **Load user sources** (personalized URLs like `https://openai.com/blog`) 5. **Resolve LLM provider** — decrypt user's API key, create provider instance (`Arc`) 6. **Resolve models** — research model + writing model (user override or admin default) 7. **Setup rate limiter** — per-user or global provider limiter 8. **Prepare LLM scraping option** — if `use_llm_for_article_extraction` enabled, clone provider+model for concurrent use 9. **Initialize tracking structures** — `filled_counts` (per-category article count), `all_scraped` (category→articles), `all_overflow` (dropped overflow), `seen_urls` (cross-phase dedup), classification categories (user categories + "Autre") --- ## Phase 1: Personalized Sources **Skipped entirely if user has 0 sources.** ### 1a. Extract article links from source pages - For each source (max 10), fetch the source page HTML - If `use_llm_for_source_links` enabled: send HTML `` + first 8000 chars of `` to LLM → extract article URLs (falls back to heuristic if LLM fails) - Otherwise: parse HTML `` links, filter by same-domain, non-homepage path, exclude `/tag/`, `/login/`, static assets, etc. - Over-fetch: `2 × max_articles_per_source` candidates per source - Deduplicate candidate URLs ### 1b. Scrape candidate articles - Fetch each URL (bounded concurrency: 5 with LLM extraction, 10 without) - SSRF check (no private IPs), 15s timeout, 5MB body limit - If `use_llm_for_article_extraction` enabled: send `` + body text to LLM → extract title, date, body, error detection (falls back to heuristic if LLM fails) - Otherwise: HTML parsing heuristics for title (``, `og:title`), date (meta tags, JSON-LD, `<time>`), body (strip scripts/nav), soft-404 detection - Capture final URL after redirects (canonical URL) ### 1c. Filter empty content - Remove articles where scraped body text is empty (scrape failure, soft 404, too old) - Trace dropped articles in `article_history` with `status: filtered_empty` ### 1d. Filter against article history - Hash each URL (normalized: lowercase, strip fragments/UTM params/trailing slashes) - Query `article_history` for existing hashes → remove matches - Trace dropped articles with `status: filtered_history` ### 1e. Retry if under-filled - If valid articles < `categories × max_items_per_category` and history is enabled - Re-scrape source pages for NEW links (exclude already-fetched URLs) - Scrape + filter empty + filter history on retry candidates - Merge with existing valid articles - Only 1 retry attempt ### 1f. LLM classification - Send articles (title + first 500 chars of body) + categories + "Autre" to LLM - LLM returns `{assignments: [{index, category}]}` mapping each article to a category - Overflow: articles that exceed both target category AND "Autre" limits → collected in `all_overflow` - **LLM call logged** with full prompt/response/timing ### 1g. Enforce source diversity - Count domains across all categories - Remove articles where domain exceeds `max_articles_per_source` - Trace dropped articles with `status: filtered_diversity` - Recount category fill levels --- ## Phase 2: Web Search Fallback **Skipped if all user-defined categories are already filled to `max_items_per_category`.** ### 2a. Compute category gaps - For each user category: `needed = max - already_filled` - Only proceed if any category needs more ### 2b. Load recent domains for diversity - If `source_diversity_window > 0`: extract domains from last N syntheses' JSONB sections - Used as soft "avoid if possible" instruction in search prompt ### 2c. LLM web search pass - Build search prompt with theme, categories, gap counts ("find N articles for AI News, M for Cybersecurity"), recent domains to avoid, personalized source URLs - Call `provider.generate_search_pass()` with web search tool enabled - **LLM call logged** with full prompt/response/timing - Returns structured JSON: `{category_0: [{title, url, summary}], category_1: [...]}` ### 2d. Filter pipeline on search results - **Parse** LLM output into `(category_key, Vec<NewsItem>)` - **Filter homepage URLs** — drop articles with path `/` or empty - **Cross-phase dedup** — drop URLs already seen in Phase 1 - **Dedup by URL** — drop duplicate URLs within Phase 2 (case-insensitive) - **Limit articles per source** — enforce `max_articles_per_source` per domain (spread across categories first, then fill) - **Filter against article history** — BEFORE scraping (saves HTTP requests), drop already-seen URLs - Each drop traced in `article_history` with appropriate status ### 2e. Scrape web search results - Same scraping as Phase 1 (bounded concurrency, SSRF check, optional LLM extraction) - Filter empty content (scrape failures, soft 404, too old) - Trace drops ### 2f. LLM classification - Same as Phase 1 classification but with Phase 2 articles - `filled_counts` carries over from Phase 1 — categories already partially filled - Overflow collected - **LLM call logged** - Merge results into `all_scraped` --- ## "Autre" Fill-Up - Count total articles across all categories - Target = `75% × (categories × max_items_per_category)` (user categories only, "Autre" excluded from denominator) - If shortfall > 0 and overflow exists: - For each overflow article: check if domain is under `max_articles_per_source` limit - Add to `all_scraped["category_autre"]` up to the shortfall --- ## Combined Rewrite Pass - **Fail if no articles** — return error if all categories are empty - **Build rewrite prompt** — serialize all scraped articles with body content, instruct LLM to rewrite title + summary (4-5 lines) faithfully based on scraped content - **Build rewrite schema** — `minItems`/`maxItems` set to ACTUAL count per category (not user max), empty categories omitted, "Autre" included if non-empty - **LLM rewrite pass** — call `provider.generate_rewrite_pass()` with writing model - **LLM call logged** with full prompt/response/timing - **Build final sections** — map `category_N` keys to user category names, add "Autre" section if present, omit empty categories - **Restore scraped URLs** — replace any hallucinated URLs from LLM rewrite with the validated scraped URLs (matched by category + position) --- ## Save + Record - **Sanitize** — strip `\u0000` null bytes from JSON (PostgreSQL rejects them in JSONB) - **Save synthesis** — insert into `syntheses` table with `job_id`, `week` (ISO week), `sections` (JSONB), `status: completed` - **Record used articles** — insert each article URL into `article_history` with `status: used`, `synthesis_id`, `job_id`, and category name (for future dedup + provenance) --- ## Summary of LLM Calls (up to 4 per generation) | # | Call | When | Model | |---|---|---|---| | 1 | Classification Phase 1 | After Phase 1 scraping | research | | 2 | Web Search | Phase 2 start | research | | 3 | Classification Phase 2 | After Phase 2 scraping | research | | 4 | Rewrite | After both phases | writing | Plus optionally per-article calls for LLM link extraction and LLM article extraction (when those settings are enabled). ## Summary of Filtering Steps | Step | Phase | What's dropped | |---|---|---| | Empty content | 1 & 2 | Scrape failures, soft 404s, too old | | Article history | 1 & 2 | Already used in previous syntheses | | Homepage URLs | 2 | Path is `/` or empty | | Cross-phase dedup | 2 | URLs already found in Phase 1 | | URL dedup | 2 | Duplicate URLs within Phase 2 | | Source diversity | 1 & 2 | Domain exceeds `max_articles_per_source` | | Category overflow | 1 & 2 | Category + "Autre" both full |