Fix rate limiter bug, simplify v2 code
Bug fix: - Per-generation rate limiter was creating a new instance on every check, making user rate limit overrides non-functional. Fixed by creating the limiter once at pipeline start and reusing for both passes. Simplifications: - Extract spawn_task closure in scrape_articles (deduplicate spawn blocks) - Use idiomatic if let Ok(...) instead of if let Some(..).ok() in scraper - Replace manual loop with iterator chain in export_keys handler - Simplify check_rate_limit to single boolean check - Simplify handleImport settings merge (spread already provides defaults) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>master
parent
0f66c28c38
commit
98528f51bd
@ -0,0 +1,230 @@
|
||||
# Functional & UX Specification: ai-weekly-synth v1 → v2
|
||||
|
||||
This document describes the v1-to-v2 changes from a user and product perspective. It covers what changes, why, and how the user experiences it — without referencing implementation details.
|
||||
|
||||
---
|
||||
|
||||
## 1. Data model: dynamic sections replace fixed categories
|
||||
|
||||
### What changes
|
||||
|
||||
v1 stores each synthesis using 5 hardcoded category slots (major announcements, financial sector, other enterprises, public sector, general public). v2 replaces these with a single ordered list of sections, each with a user-defined title.
|
||||
|
||||
### Why
|
||||
|
||||
The fixed categories constrained the application to a single topic structure. With dynamic sections, the synthesis adapts to whatever categories the user has configured in Settings — whether that is 2 categories or 15, and regardless of their names.
|
||||
|
||||
### User-visible impact
|
||||
|
||||
- New syntheses are generated using only the dynamic sections structure.
|
||||
- Old syntheses that were stored with the fixed category fields will no longer display their content. Instead, a message "Aucune section trouvée dans cette synthèse." appears in the detail view and in emails. This is a **breaking change** for legacy data.
|
||||
- The home page card preview for legacy syntheses will show "Aucune annonce majeure cette semaine." instead of the first category's items.
|
||||
|
||||
---
|
||||
|
||||
## 2. Home page
|
||||
|
||||
### Title wording
|
||||
|
||||
The page heading changes from "Synthèses d'Actualités **IA**" to "Synthèses d'Actualités **par IA**" — a minor phrasing adjustment that clarifies the syntheses are made *by* AI, not about AI (the topic is configurable).
|
||||
|
||||
---
|
||||
|
||||
## 3. Settings page
|
||||
|
||||
The Settings page gains significant new functionality. The existing controls (theme, max age, max items per category, categories list) remain unchanged. All additions are described below in the order they appear on the page.
|
||||
|
||||
### 3.1 Header: Export / Import buttons
|
||||
|
||||
Two new buttons appear in the page header, to the right of the title:
|
||||
|
||||
| Button | Icon | Behavior |
|
||||
|--------|------|----------|
|
||||
| **Exporter** | Download arrow | Downloads the current settings as a `settings.json` file |
|
||||
| **Importer** | Upload arrow | Opens a file picker (`.json` only). Loads the selected file's contents into the form. A success banner reminds the user to click "Enregistrer" to persist the imported settings. Invalid JSON shows an error banner. |
|
||||
|
||||
Import merges the uploaded values over the defaults, so missing fields fall back to their default values.
|
||||
|
||||
### 3.2 Search agent behavior (new field)
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Label** | Comportement de l'agent de recherche |
|
||||
| **Control** | Multi-line text area (3 rows) |
|
||||
| **Default** | "Tu peux également utiliser d'autres sources pertinentes trouvées via la recherche Google." |
|
||||
| **Help text** | Personnalisez les instructions données à l'IA concernant sa méthode de recherche. |
|
||||
|
||||
This lets the user customize the instructions injected into the AI research prompt — for example, restricting the AI to only the user's custom sources, or asking it to prioritize certain types of sites.
|
||||
|
||||
### 3.3 AI model for research (new field)
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Label** | Modèle d'IA (Recherche et Extraction) |
|
||||
| **Control** | Dropdown |
|
||||
| **Default** | Gemini 3.1 Pro |
|
||||
| **Help text** | Choisissez le modèle d'IA utilisé pour rechercher et extraire les informations. Le modèle Pro est conseillé pour des résultats plus pertinents et une meilleure analyse. |
|
||||
|
||||
Available options:
|
||||
|
||||
| Display label | Model ID |
|
||||
|---------------|----------|
|
||||
| Gemini 3.1 Pro (conseillé) | gemini-3.1-pro-preview |
|
||||
| Gemini 3.0 Flash | gemini-3-flash-preview |
|
||||
| Gemini 3.1 Flash Lite | gemini-3.1-flash-lite-preview |
|
||||
| Gemini 2.5 Flash | gemini-2.5-flash |
|
||||
|
||||
This model is used for Stage 1 of synthesis generation (searching and collecting news articles).
|
||||
|
||||
### 3.4 AI model for writing (new field)
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Label** | Modèle d'IA (Rédaction et Synthèse) |
|
||||
| **Control** | Dropdown |
|
||||
| **Default** | Gemini 3.1 Pro |
|
||||
| **Help text** | Choisissez le modèle d'IA utilisé pour le second agent, chargé de rédiger et structurer la synthèse finale. |
|
||||
|
||||
Same 4 model options as above. This model is used for Stage 2 (rewriting titles and summaries from scraped content).
|
||||
|
||||
Users can mix models — for example, use a faster/cheaper model for research and a higher-quality model for final writing, or vice versa.
|
||||
|
||||
### 3.5 Rate limiting (new section)
|
||||
|
||||
A new section titled **"Limitation de taux (Rate Limiting)"** appears below the categories list, visually separated by a horizontal rule.
|
||||
|
||||
| Field | Label | Control | Constraints | Default |
|
||||
|-------|-------|---------|-------------|---------|
|
||||
| Max requests | Requêtes maximum | Number input | min: 1 | 29 |
|
||||
| Time window | Fenêtre de temps (ms) | Number input | min: 1000, step: 1000 | 60000 |
|
||||
|
||||
**Help text:** Configurez le nombre maximum de requêtes autorisées vers l'API Gemini pendant la fenêtre de temps spécifiée (en millisecondes). Par défaut : 29 requêtes par minute (60000 ms).
|
||||
|
||||
This gives users control over API throttling — useful for users on free-tier API plans with stricter quotas, or users who want to increase throughput on paid plans.
|
||||
|
||||
### 3.6 Danger zone (new section)
|
||||
|
||||
A red-themed section titled **"Zone de danger"** (with a warning triangle icon) appears at the bottom of the form, separated by a red horizontal rule.
|
||||
|
||||
It contains a single action:
|
||||
|
||||
**Nettoyer les données**
|
||||
- Description: "Cette action supprimera définitivement toutes vos synthèses générées et tous vos articles/sources enregistrés. Vos paramètres de configuration seront conservés."
|
||||
- A "Supprimer les données" button initiates the flow.
|
||||
- Clicking it reveals a two-button confirmation state: "Confirmer la suppression" and "Annuler".
|
||||
- During deletion, the confirm button shows a loading state ("Suppression...") and both buttons are disabled.
|
||||
- On success, a green banner confirms the deletion. On failure, an error banner appears.
|
||||
- Only the current user's data is affected. Settings are preserved.
|
||||
|
||||
---
|
||||
|
||||
## 4. Sources page
|
||||
|
||||
### 4.1 CSV Import / Export (new section)
|
||||
|
||||
A new card titled **"Import / Export CSV"** is inserted between the existing "Ajouter une source" form and the "Import en masse" text area.
|
||||
|
||||
**Description:** "Sauvegardez vos sources ou importez-en de nouvelles depuis un fichier CSV."
|
||||
|
||||
Two buttons appear side by side:
|
||||
|
||||
| Button | Icon | Behavior |
|
||||
|--------|------|----------|
|
||||
| **Exporter en CSV** | Download arrow | Downloads all current sources as a `sources.csv` file with a `Titre,URL` header row. Values are quoted and double-quotes are escaped. |
|
||||
| **Importer depuis un CSV** | Upload arrow | Opens a file picker (`.csv` only). Parses the file and bulk-creates all valid sources. |
|
||||
|
||||
**CSV import behavior:**
|
||||
- Accepts both comma (`,`) and semicolon (`;`) as delimiters (auto-detected per line).
|
||||
- Automatically skips the first row if it looks like a header (contains "titre" or "title", case-insensitive).
|
||||
- Handles quoted fields and escaped double-quotes (`""`).
|
||||
- Automatically prepends `https://` to URLs missing a protocol.
|
||||
- If no valid sources are found, an error message is shown below the buttons.
|
||||
- Importing is additive — existing sources are not deleted.
|
||||
|
||||
The existing "Import en masse" text area (semicolon-separated, one per line) remains available below the CSV section as an alternative bulk import method.
|
||||
|
||||
---
|
||||
|
||||
## 5. Synthesis generation quality improvements
|
||||
|
||||
These changes affect the output quality of generated syntheses. They are not directly visible as UI controls but improve the user's experience when reading syntheses.
|
||||
|
||||
### 5.1 Deep-link URL enforcement
|
||||
|
||||
**Problem in v1:** The AI sometimes returned homepage URLs (e.g., `https://lemonde.fr`) instead of direct article links.
|
||||
|
||||
**v2 behavior:**
|
||||
- The AI prompt now explicitly instructs the model to always provide deep-link URLs pointing to specific articles, and never homepage/root URLs.
|
||||
- The system instruction reinforces this requirement.
|
||||
- During validation, any URL whose path is `/` or empty is automatically rejected before content is scraped.
|
||||
|
||||
### 5.2 Enhanced broken link detection
|
||||
|
||||
**Problem in v1:** Some URLs returned by the AI were broken (soft 404 pages that return HTTP 200). v1 only detected errors by checking the page title and H1 for keywords like "404" or "not found".
|
||||
|
||||
**v2 adds three additional detection layers:**
|
||||
|
||||
1. **Redirect/canonical URL analysis** — If the page was redirected (via the proxy's final URL), or has a canonical or og:url meta tag that points to a known error path (`/404`, `/404.html`, `/error`), the article is rejected.
|
||||
|
||||
2. **Short-page body text analysis** — For pages with less than 1500 characters of body text (typically error pages), the full body text is checked for error phrases in English and French ("page not found", "page introuvable", "could not be found", "the requested page", or combinations of "404" with "not found"/"error").
|
||||
|
||||
3. **The existing detection** (title/H1 keyword matching, `noindex` meta tag) remains in place.
|
||||
|
||||
The combined effect is significantly fewer broken links in the final synthesis.
|
||||
|
||||
### 5.3 Original title preservation
|
||||
|
||||
**Problem in v1:** Article titles in the synthesis were generated entirely by the AI based on scraped content, which could result in mistranslated or inaccurate titles.
|
||||
|
||||
**v2 behavior:**
|
||||
- During scraping, the original title is extracted from each article page. The extraction uses the following priority: `og:title` meta tag, then the first `<h1>` element, then the document `<title>`, and finally the AI-provided title as a fallback.
|
||||
- The original title is passed to the writing AI along with the scraped content.
|
||||
- The writing AI is instructed to use the original title as a base, with these language rules:
|
||||
- English titles must be kept in English (not translated).
|
||||
- French titles must be kept in French.
|
||||
- Titles in other languages must be translated into French.
|
||||
|
||||
### 5.4 Longer summaries
|
||||
|
||||
Article summaries increase from 3-4 lines (v1) to 4-5 lines (v2), providing slightly more detailed coverage of each article.
|
||||
|
||||
### 5.5 Configurable rate limiting at runtime
|
||||
|
||||
The rate limiter (which throttles requests to the Gemini API during synthesis generation) now reads its configuration from the user's settings at the start of each generation. In v1, these values were hardcoded at 29 requests per 60 seconds.
|
||||
|
||||
---
|
||||
|
||||
## 6. Synthesis detail page
|
||||
|
||||
### 6.1 Legacy data handling
|
||||
|
||||
When viewing a synthesis that has no `sections` data (i.e., a v1-era synthesis stored with the old fixed-category fields):
|
||||
|
||||
| Area | v1 behavior | v2 behavior |
|
||||
|------|-------------|-------------|
|
||||
| **Page rendering** | Falls back to rendering the 5 hardcoded category sections | Shows an italic centered message: "Aucune section trouvée dans cette synthèse." |
|
||||
| **Email body** | Falls back to listing the 5 hardcoded categories | Includes the text: "Aucune section trouvée dans cette synthèse." |
|
||||
|
||||
This is an intentional simplification. The old hardcoded field names are no longer referenced anywhere in v2.
|
||||
|
||||
---
|
||||
|
||||
## 7. Firestore security rules
|
||||
|
||||
The settings validation rules are updated to accept the new optional fields (`aiModel`, `secondaryAiModel`, `rateLimitMaxRequests`, `rateLimitTimeWindowMs`). All four are optional — existing settings documents without them remain valid and writable.
|
||||
|
||||
- `aiModel` and `secondaryAiModel` must be strings if present.
|
||||
- `rateLimitMaxRequests` and `rateLimitTimeWindowMs` must be positive numbers if present.
|
||||
|
||||
---
|
||||
|
||||
## Summary of changes by page
|
||||
|
||||
| Page | Changes |
|
||||
|------|---------|
|
||||
| **Home** | Title wording ("IA" → "par IA"). Legacy synthesis cards show empty preview instead of old category data. |
|
||||
| **Settings** | New: Export/Import buttons, search agent behavior textarea, 2 AI model dropdowns, rate limiting controls, danger zone with data clearing. |
|
||||
| **Sources** | New: CSV export button, CSV import via file picker. Existing bulk text import unchanged. |
|
||||
| **Synthesis Detail** | Legacy syntheses show "no sections found" instead of the 5 hardcoded categories. |
|
||||
| **Generate Synthesis** | No UI changes. Backend improvements: better URL quality, original title preservation, longer summaries, configurable models. |
|
||||
| **Login** | No changes. |
|
||||
Loading…
Reference in New Issue