Many modern sites (Hugo, WordPress, Next.js) load articles via JavaScript
but include full article URLs in JSON-LD schema.org markup in the <head>.
The scraper now extracts these first (highest quality), then falls back
to <a href> heuristic extraction. Supports ItemList, BlogPosting,
NewsArticle, @graph arrays, and mainEntity wrappers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Date parser now supports: 25/03/2026, 25-03-2026, March 25 2026,
25 mars 2026, 15 février 2026, and short month variants.
Articles without dates are no longer routed to a separate category —
they stay in their LLM-classified category with date shown as empty.
This prevents losing good articles in a catch-all section.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Date aligned left, source URL aligned right. URL stripped of protocol
and truncated to 40 chars with "..." if too long. Full URL on hover.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The API client expects empty responses to use 204, not 200.
Returning 200 with no body caused JSON parse error in the frontend.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract content settings card and sources card into dedicated components,
reducing ThemeManager from 938 to 233 lines while keeping theme list CRUD
and selector in the parent.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace raw fetch in handleStop with synthesesApi.stop()
- Add stop() method to synthesesApi
- Replace raw <button> elements in GenerateSynthesis with Button component (generate, retry, stop)
- Remove deprecated LLM link extraction schema reference from technical_specs.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- export_csv now accepts optional theme_id query param and filters accordingly
- Add UpdateThemeRequest::validate() with bounds checking; call it in the update handler
- Verify theme ownership in sources::create when theme_id is provided
- Update STATUS_OPTIONS (add filtered_too_old, filtered_not_article; remove filtered_duplicate) and SOURCE_TYPE_OPTIONS (add brave_search; remove overflow) in ArticleHistory
- Replace hardcoded French strings ('Confirmer', 'Erreur inconnue') with t() calls; add settings.apiKeys.unknownError key to fr.ts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Bulk/CSV import now passes theme_id through to DB
- Preferred source update scoped by theme_id (no cross-theme reset)
- Theme creation sends sensible defaults from frontend
- Scheduler wraps generation in 15-minute timeout
- Job store cleanup runs every 5 minutes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Trim architecture.md significantly (section 1 overview, technology stack, deployment topology,
module inventory lists, LLM trait block, pipeline details, data model table, full API tables,
background task list). Replace section 5 API tables with a one-liner. Requirements.md sections
3.1/3.5/3.6/3.7/3.8 and 4.2 condensed with cross-references. deployment.md security feature
list replaced by cross-reference to architecture.md Section 6. functional_specs.md Section 3
gains a cross-reference to technical_specs.md Section 5.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The full pipeline algorithm is now in technical_specs.md with added
details: preferred source ordering, windowed waves, is_article filter,
date fallback, "Articles sans date" category, cancellation support.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaced by 7 consolidated docs: requirements, functional_specs,
architecture, technical_specs, dev_guidelines, qa_guidelines, deployment.
algorithm.md retained as pipeline reference.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Updated project structure, features list, documentation links, database
tables, testing commands, and environment variables to reflect the
current state (multi-theme, scheduling, preferred sources, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace "Lire la suite" and "Reduire" with t('synthesis.readMore')
and t('synthesis.collapse'). Adds both keys to fr.ts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Moves JobEntry, JobStore, ProgressEvent, JOB_TTL, and emit_progress
to a dedicated module. Updates imports in synthesis.rs, generation.rs,
scheduler.rs, and app_state.rs. synthesis.rs re-exports for backward
compatibility.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces ~200 duplicated lines in Phase 1 (personalized sources) and
Phase 2 (Brave Search) with a shared scrape_and_classify_batch function.
Uses ScrapeClassifyCtx to bundle shared parameters. Also prepares
synthesis.rs for JobStore extraction by re-exporting from job_store.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
normalizeUrl and isValidUrl are now in ~/utils/url. ThemeManager
and sources-utils tests import from the new location. The /sources
route redirect to /themes is preserved in App.tsx.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Settings tests now reference fields that exist in the current UI
(max articles per source instead of removed theme field). Generate
tests now mock the themes API and use current pipeline step names.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The live generation test depends on real OpenAI + web scraping.
Adding a second source improves chances of finding articles.
Retries increased from 1 to 2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
article_history_days=0 disables "used" trace entries, so the test
found 0 entries. Changed to 90 to enable tracing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Spawns a tokio task that checks for due schedules every 60 seconds,
runs generation via run_generation_inner, and sends emails to configured
recipients before marking each schedule as run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Users can mark sources as preferred via star buttons on the theme page.
Preferred sources are processed first in the pipeline (ordered before
non-preferred in waves, shuffled separately then merged).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- settings.spec.ts: test batch_size instead of removed theme field
- settings-export.spec.ts: test batch_size export/import, open collapsed details section
- sources.spec.ts: convert to API-based test since sources now belong to themes
- generation-live.spec.ts: already updated (no changes needed)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds Arc<AtomicBool> cancellation flag to JobStore/JobEntry. The pipeline
checks the flag before each wave and after each batch, then saves whatever
articles have been collected. A new POST /syntheses/generate/:job_id/stop
endpoint sets the flag. The frontend shows a red stop button during generation
and POSTs to the stop endpoint on click.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Diversity filter works across batches (source_counts updated after classify).
With batch_size=5, all 3 articles fit in one batch, bypassing the filter.
batch_size=1 forces per-article processing so the filter triggers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without a date, articles are routed to "Articles sans date" instead
of their classified category, breaking pipeline tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
