`, `og:title`), date (meta tags, JSON-LD, `<time>`), body (strip scripts/nav), soft-404 detection - If article scraped body text is empty (scrape failure, soft 404, too old): - Trace dropped articles in `article_history` with `status: filtered_empty` - Move to next url - Send article (title + first 500 chars of body) + categories + "Autre" to LLM. LLM returns `{title, summary, category}` mapping the article to a category. The LLM generates the summary and a also a title if the provided title is empty - **LLM call logged** with full prompt/response/timing - Add the article to `article_scraped` and increase `filled_counts` - if number of articles in the category of this artcile exceeds `max_items_per_category`: change the article catgeory to "Autre" - If the total number of articles in `article_scraped` exceeds `number of categories (including Autre) × max_items_per_category` then exit for loop and move to synthesis generation --- ## Phase 2: Web Search Fallback **Skipped if all user-defined categories are already filled to `max_items_per_category`.** ### 2a. Compute category gaps - For each user category: `needed = max - already_filled` - Only proceed if any category needs more ### 2b. LLM web search pass - Build search prompt with theme, categories, gap counts ("find N articles for category_1, M for category_2") - Send search prompt to LLM. LLM returns structured JSON: `{category_0: [{title, url, summary}], category_1: [...]}` - **LLM call logged** with full prompt/response/timing - **Filter homepage URLs** — drop articles with path `/` or empty - **Cross-phase dedup** — drop URLs already seen in Phase 1 - **Dedup by URL** — drop duplicate URLs within Phase 2 (case-insensitive) - **Limit articles per source** — enforce `max_articles_per_source` per domain (spread across categories first, then fill) - **Filter against article history** — BEFORE scraping (saves HTTP requests), drop already-seen URLs - Each drop traced in `article_history` with appropriate status ### 2c. Scrape web search results - Same scraping as Phase 1 (bounded concurrency, SSRF check, optional LLM extraction) - Filter empty content (scrape failures, soft 404, too old) - Trace drops - Merge results into `all_scraped` - Move to synthesis generation --- ## Save + Record - **Sanitize** — strip `\u0000` null bytes from JSON (PostgreSQL rejects them in JSONB) - **Save synthesis** — insert into `syntheses` table with `job_id`, `week` (ISO week), `sections` (JSONB), `status: completed` - **Record used articles** — insert each article URL into `article_history` with `status: used`, `synthesis_id`, `job_id`, and category name (for future dedup + provenance)

# Synthesis Generation Pipeline — Full Algorithm ## Initialization 1. **Load user settings** from DB (categories, provider, models, max_items, etc.) 2. **Cleanup** — delete old article history entries (>N days, dropped only) + truncate old LLM call logs 3. **Validate** — if no categories configured, there will just be the default category "Autre". 4. **Load user sources** (personalized URLs like `https://openai.com/blog`) 5. **Resolve LLM provider** — decrypt user's API key, create provider instance (`Arc`) 6. **Resolve models** — research model + writing model (user override or admin default) 7. **Setup rate limiter** — per-user or global provider limiter 8. **Prepare LLM scraping option** — if `use_llm_for_article_extraction` enabled, clone provider+model for concurrent use 9. **Initialize tracking structures** — `article_scraped` (category→articles), `source_counts` (per-source article count), `url_soucre` (per-article source), `filled_counts` (per-category article count), `seen_urls` (cross-phase dedup), classification categories (user categories + "Autre") --- ## Phase 1: Personalized Sources **Skipped entirely if user has 0 sources.** ### 1a. Extract article links from source pages and filter against article history - Query `article_history` for the last source used. Reorder the personalized source so that the first source is the one following the last source used (rolling window) - For each source, fetch the source page HTML: - If `use_llm_for_source_links` enabled: send HTML `` + first 8000 chars of `` to LLM → extract all article URLs up to a maximum of 10, with the most recent first. If LLM call fails, fall back to HTML parsing as described below. - **LLM call logged** with full prompt/response/timing - Otherwise: parse HTML `` links, filter by same-domain, non-homepage path, exclude `/tag/`, `/login/`, `/contact/`,`/presentation/`,`/newsletter/`, static assets, etc. and keep only the first 10 links found - Deduplicate candidate URLs - Hash each URL (normalized: lowercase, strip fragments/UTM params/trailing slashes) - Query `article_history` for existing hashes → remove matches - Trace dropped articles with `status: filtered_history` - Add the url to `url_soucre` ### 1b. Scrape, classify and summarize articles - For each url from step 1a: - if the number of articles in `source_counts` for the source of the current url exceeds `max_articles_per_source`: - Trace dropped article with `status: filtered_diversity` - Move to next url - Fetch each URL (bounded concurrency: 5 with LLM extraction, 10 without). - SSRF check (no private IPs), 15s timeout, 5MB body limit. - HTML parsing heuristics for title (``, `og:title`), date (meta tags, JSON-LD, `<time>`), body (strip scripts/nav), soft-404 detection - If article scraped body text is empty (scrape failure, soft 404, too old): - Trace dropped articles in `article_history` with `status: filtered_empty` - Move to next url - Send article (title + first 500 chars of body) + categories + "Autre" to LLM. LLM returns `{title, summary, category}` mapping the article to a category. The LLM generates the summary and a also a title if the provided title is empty - **LLM call logged** with full prompt/response/timing - Add the article to `article_scraped` and increase `filled_counts` - if number of articles in the category of this artcile exceeds `max_items_per_category`: change the article catgeory to "Autre" - If the total number of articles in `article_scraped` exceeds `number of categories (including Autre) × max_items_per_category` then exit for loop and move to synthesis generation --- ## Phase 2: Web Search Fallback **Skipped if all user-defined categories are already filled to `max_items_per_category`.** ### 2a. Compute category gaps - For each user category: `needed = max - already_filled` - Only proceed if any category needs more ### 2b. LLM web search pass - Build search prompt with theme, categories, gap counts ("find N articles for category_1, M for category_2") - Send search prompt to LLM. LLM returns structured JSON: `{category_0: [{title, url, summary}], category_1: [...]}` - **LLM call logged** with full prompt/response/timing - **Filter homepage URLs** — drop articles with path `/` or empty - **Cross-phase dedup** — drop URLs already seen in Phase 1 - **Dedup by URL** — drop duplicate URLs within Phase 2 (case-insensitive) - **Limit articles per source** — enforce `max_articles_per_source` per domain (spread across categories first, then fill) - **Filter against article history** — BEFORE scraping (saves HTTP requests), drop already-seen URLs - Each drop traced in `article_history` with appropriate status ### 2c. Scrape web search results - Same scraping as Phase 1 (bounded concurrency, SSRF check, optional LLM extraction) - Filter empty content (scrape failures, soft 404, too old) - Trace drops - Merge results into `all_scraped` - Move to synthesis generation --- ## Save + Record - **Sanitize** — strip `\u0000` null bytes from JSON (PostgreSQL rejects them in JSONB) - **Save synthesis** — insert into `syntheses` table with `job_id`, `week` (ISO week), `sections` (JSONB), `status: completed` - **Record used articles** — insert each article URL into `article_history` with `status: used`, `synthesis_id`, `job_id`, and category name (for future dedup + provenance)