Add v2 changes design spec
8 changes covering: settings export/import, dual model selection, user rate limit overrides, original title preservation, enhanced broken link detection, deep-link URL enforcement, settings schema expansion, and empty sections fallback message. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>master
parent
62c2d959d3
commit
74e2cb0273
@ -0,0 +1,289 @@
|
|||||||
|
# Design: AI Weekly Synth v2 Changes
|
||||||
|
|
||||||
|
**Date**: 2026-03-21
|
||||||
|
**Source**: docs/functional-spec-v2.md
|
||||||
|
**Scope**: 8 changes to implement on the existing Rust/SolidJS codebase
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The v2 functional spec describes improvements to the original React/Firebase app. The codebase has already been rewritten to Rust (Axum) + SolidJS + Postgres. Several v2 features were implemented during the rewrite (dynamic sections, "par IA" title, search agent behavior, CSV import/export, 4-5 line summaries). This design covers the remaining gaps.
|
||||||
|
|
||||||
|
**Dropped from v2 spec:**
|
||||||
|
- Danger Zone (Section 3.6) — per-item delete is sufficient; bulk delete is high risk, low convenience
|
||||||
|
- Firestore rules (Section 7) — N/A, we use Postgres
|
||||||
|
- Cross-provider model mixing — single provider with two model selections instead
|
||||||
|
|
||||||
|
**Spec divergences (design-time enhancements):**
|
||||||
|
- Settings export can optionally include API keys (not in functional spec, added for self-hosted backup use case)
|
||||||
|
- Rate limit overrides are nullable (use admin defaults when unset) rather than hardcoded 29/60000
|
||||||
|
- Title extraction priority is `<title>` -> `og:title` -> `<h1>` -> AI fallback (per project owner preference, differs from functional spec)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change 1: Settings Export/Import
|
||||||
|
|
||||||
|
### What
|
||||||
|
Two buttons in the Settings page header: "Exporter" (download JSON) and "Importer" (upload JSON). A checkbox "Inclure les cles API" controls whether decrypted API keys are included in the export.
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
New endpoint for decrypted key export:
|
||||||
|
|
||||||
|
- `POST /api/v1/user/api-keys/export` — returns decrypted keys as `[{ provider_name, api_key }]`. Uses POST (not GET) to require CSRF token. Rate-limited to 3 calls per minute. Audit-logged.
|
||||||
|
|
||||||
|
Import uses the existing `PUT /api/v1/settings` (updated in Change 2 to accept new fields) and `POST /api/v1/user/api-keys` per key.
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
- Download and Upload icons in Settings header
|
||||||
|
- Checkbox: "Inclure les cles API" with warning: "Les cles API seront incluses en clair dans le fichier. Ne partagez pas ce fichier."
|
||||||
|
- On export: fetch settings + optionally POST to export keys, merge into JSON, trigger download as `settings.json`
|
||||||
|
- On import: parse JSON, validate shape, populate form + call API key endpoints, show success banner ("N'oubliez pas d'enregistrer")
|
||||||
|
- Missing fields fall back to defaults (merge over `DEFAULT_SETTINGS`)
|
||||||
|
|
||||||
|
**Dependency:** Requires Change 2 (new settings fields) to be implemented first so the import can handle all fields.
|
||||||
|
|
||||||
|
### Files to modify
|
||||||
|
- `backend/src/handlers/api_keys.rs` — add `export_keys` handler
|
||||||
|
- `backend/src/router.rs` — add route
|
||||||
|
- `frontend/src/pages/Settings.tsx` — add export/import UI
|
||||||
|
- `frontend/src/api/apiKeys.ts` — add `exportKeys()` method
|
||||||
|
- `frontend/src/i18n/fr.ts` — add translation keys
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change 2: Provider/Model Selection + Two Model Dropdowns
|
||||||
|
|
||||||
|
### What
|
||||||
|
The backend `settings` table currently has NO `ai_provider` or `ai_model` columns — the frontend types have them but they aren't persisted. This change adds provider/model persistence plus a second model for writing.
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
|
||||||
|
**Migration** (`20260321000010_add_settings_v2_fields.sql`):
|
||||||
|
```sql
|
||||||
|
ALTER TABLE settings
|
||||||
|
ADD COLUMN ai_provider VARCHAR(100) NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN ai_model VARCHAR(100) NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN ai_model_writing VARCHAR(100) NOT NULL DEFAULT '';
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: table is called `settings` (not `user_settings`).
|
||||||
|
|
||||||
|
**Model changes** (`models/settings.rs`):
|
||||||
|
- Add `ai_provider`, `ai_model`, `ai_model_writing` to `UserSettings`, `SettingsResponse`, `UpdateSettingsRequest`
|
||||||
|
- Validation: if `ai_model_writing` is empty, fall back to `ai_model` at generation time
|
||||||
|
|
||||||
|
**DB changes** (`db/settings.rs`):
|
||||||
|
- Update `get_or_create_default` and `upsert` queries to include new columns
|
||||||
|
|
||||||
|
**Pipeline changes** (`services/synthesis.rs`):
|
||||||
|
- Update `resolve_model()` to read user's `ai_provider` and `ai_model` from settings first, fall back to admin default model only if user hasn't selected one
|
||||||
|
- Pass 1 uses `ai_model`, Pass 2 uses `ai_model_writing` (or `ai_model` if writing model is empty)
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
- Settings page: single provider dropdown + two model dropdowns
|
||||||
|
- "Modele d'IA (Recherche et Extraction)" — uses `ai_model`
|
||||||
|
- "Modele d'IA (Redaction et Synthese)" — uses `ai_model_writing`
|
||||||
|
- Both model dropdowns filtered by selected provider
|
||||||
|
- If only one provider configured, hide provider dropdown
|
||||||
|
- Help text per spec sections 3.3 and 3.4
|
||||||
|
|
||||||
|
### Files to modify
|
||||||
|
- `backend/migrations/20260321000010_add_settings_v2_fields.sql` — new migration
|
||||||
|
- `backend/src/models/settings.rs` — add 3 fields + validation
|
||||||
|
- `backend/src/db/settings.rs` — update queries
|
||||||
|
- `backend/src/handlers/settings.rs` — update response/request handling
|
||||||
|
- `backend/src/services/synthesis.rs` — use user's provider/model selection
|
||||||
|
- `frontend/src/types.ts` — already has fields, verify alignment
|
||||||
|
- `frontend/src/pages/Settings.tsx` — two model dropdowns with labels/help
|
||||||
|
- `frontend/src/i18n/fr.ts` — labels and help text
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change 3: User Rate Limit Overrides
|
||||||
|
|
||||||
|
### What
|
||||||
|
Two number fields in Settings: "Requetes maximum" and "Fenetre de temps (secondes)". Override the admin-configured defaults for the user's provider. Uses **seconds** (not milliseconds) to match the existing `admin_rate_limits.time_window_seconds` convention.
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
|
||||||
|
**Migration** (same file as Change 2):
|
||||||
|
```sql
|
||||||
|
ALTER TABLE settings
|
||||||
|
ADD COLUMN rate_limit_max_requests INTEGER,
|
||||||
|
ADD COLUMN rate_limit_time_window_seconds INTEGER;
|
||||||
|
```
|
||||||
|
|
||||||
|
Both columns are nullable — NULL means "use admin default".
|
||||||
|
|
||||||
|
**Model changes** (`models/settings.rs`):
|
||||||
|
- Add `rate_limit_max_requests: Option<i32>` and `rate_limit_time_window_seconds: Option<i32>`
|
||||||
|
- Validation: if set, max_requests >= 1, time_window_seconds >= 1
|
||||||
|
|
||||||
|
**Pipeline integration** (`services/synthesis.rs`):
|
||||||
|
- When the generation pipeline needs a rate limit slot, create a **temporary per-generation rate limiter** using the user's overrides (if set) or the admin defaults for that provider. This avoids modifying the global `ProviderRateLimiter` which is shared across all users.
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
- New "Limitation de taux" section in Settings (below categories, separated by a rule)
|
||||||
|
- Two number inputs: "Requetes maximum" (min: 1) and "Fenetre de temps (secondes)" (min: 1)
|
||||||
|
- Help text: "Configurez le nombre maximum de requetes autorisees pendant la fenetre de temps specifiee. Laissez vide pour utiliser les valeurs par defaut de l'administrateur."
|
||||||
|
- Show effective rate when values are set: "{max_requests} requetes / {time_window} secondes"
|
||||||
|
- A "Reinitialiser" link to clear overrides (sets both to null)
|
||||||
|
|
||||||
|
### Files to modify
|
||||||
|
- `backend/migrations/20260321000010_add_settings_v2_fields.sql` — add columns
|
||||||
|
- `backend/src/models/settings.rs` — add nullable fields + validation
|
||||||
|
- `backend/src/db/settings.rs` — update queries
|
||||||
|
- `backend/src/services/synthesis.rs` — per-generation rate limiter
|
||||||
|
- `frontend/src/types.ts` — add optional fields to `UserSettings`
|
||||||
|
- `frontend/src/pages/Settings.tsx` — rate limit section
|
||||||
|
- `frontend/src/i18n/fr.ts` — labels and help text
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change 4: Original Title Preservation
|
||||||
|
|
||||||
|
### What
|
||||||
|
During scraping, extract the original article title with priority: `<title>` -> `og:title` -> first `<h1>` -> AI-provided title as fallback. Pass this to the rewrite prompt so the AI preserves it, with language rules.
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
|
||||||
|
**Scraper changes** (`services/scraper.rs`):
|
||||||
|
- Refactor `extract_page_title()` to implement the full priority chain:
|
||||||
|
1. `<title>` element text (already exists)
|
||||||
|
2. `<meta property="og:title">` content attribute
|
||||||
|
3. First `<h1>` element text
|
||||||
|
4. Returns `None` if all are empty (pipeline uses AI-provided title as fallback)
|
||||||
|
|
||||||
|
**Model changes** (`models/synthesis.rs`):
|
||||||
|
- Add `original_title: String` to `ScrapedNewsItem` (populated from scraper result or AI-provided title as fallback)
|
||||||
|
|
||||||
|
**Prompt changes** (`services/prompts.rs`):
|
||||||
|
- Update `build_rewrite_prompt()` to include `original_title` per item
|
||||||
|
- Add language rules: "Utilise le titre original comme base. Les titres en anglais doivent rester en anglais. Les titres en francais doivent rester en francais. Les titres dans d'autres langues doivent etre traduits en francais."
|
||||||
|
|
||||||
|
### Files to modify
|
||||||
|
- `backend/src/services/scraper.rs` — enhanced title extraction with priority chain
|
||||||
|
- `backend/src/models/synthesis.rs` — `ScrapedNewsItem.original_title`
|
||||||
|
- `backend/src/services/prompts.rs` — rewrite prompt with title preservation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change 5: Enhanced Broken Link Detection
|
||||||
|
|
||||||
|
### What
|
||||||
|
Add two detection layers to the scraper, on top of the existing title/H1 keyword matching.
|
||||||
|
|
||||||
|
### Layer 1: Canonical/OG URL + redirect error path detection
|
||||||
|
After parsing HTML:
|
||||||
|
- Extract `<link rel="canonical" href="...">` and `<meta property="og:url" content="...">`
|
||||||
|
- Capture the final URL after redirects from `response.url()` (reqwest provides this)
|
||||||
|
- If any of these URLs' path contains `/404`, `/404.html`, `/error`, or `/not-found`, mark as soft-404
|
||||||
|
|
||||||
|
### Layer 2: Short-page body text analysis
|
||||||
|
If extracted body text has fewer than 1500 characters, scan it for error phrases:
|
||||||
|
- English: "page not found", "could not be found", "the requested page", "does not exist"
|
||||||
|
- French: "page introuvable", "page non trouvee", "n'existe pas", "n'a pas ete trouvee"
|
||||||
|
- Combined: "404" appearing near "not found" or "error" (within 50 chars)
|
||||||
|
|
||||||
|
If any match, mark as soft-404.
|
||||||
|
|
||||||
|
### Layer 3: noindex meta tag (currently missing, add it)
|
||||||
|
- Check for `<meta name="robots" content="noindex">` — if present, mark as soft-404
|
||||||
|
|
||||||
|
### Files to modify
|
||||||
|
- `backend/src/services/scraper.rs` — add `detect_canonical_error()`, `detect_short_page_error()`, `detect_noindex()`, capture final redirect URL, integrate all into `scrape_url()`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change 6: Deep-Link URL Enforcement
|
||||||
|
|
||||||
|
### What
|
||||||
|
After the LLM returns URLs in the search pass, reject any URL whose path is `/` or empty (homepage URLs). Filter before scraping.
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
- Add a filter step in `services/synthesis.rs` between parsing LLM output and scraping
|
||||||
|
- For each `NewsItem`, parse URL with `url::Url`, check if `path()` is `/` or empty
|
||||||
|
- Log rejected URLs at `warn` level, continue with remaining items
|
||||||
|
- Update search prompt in `services/prompts.rs`: add "Ne retourne JAMAIS des URLs de pages d'accueil (homepage). Fournis toujours des liens directs vers des articles specifiques."
|
||||||
|
|
||||||
|
### Files to modify
|
||||||
|
- `backend/src/services/synthesis.rs` — post-LLM URL filter
|
||||||
|
- `backend/src/services/prompts.rs` — reinforce deep-link instruction
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change 7: Database Migration
|
||||||
|
|
||||||
|
Single migration file covering Changes 2 and 3:
|
||||||
|
|
||||||
|
**File**: `backend/migrations/20260321000010_add_settings_v2_fields.sql`
|
||||||
|
|
||||||
|
```sql
|
||||||
|
ALTER TABLE settings
|
||||||
|
ADD COLUMN ai_provider VARCHAR(100) NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN ai_model VARCHAR(100) NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN ai_model_writing VARCHAR(100) NOT NULL DEFAULT '',
|
||||||
|
ADD COLUMN rate_limit_max_requests INTEGER,
|
||||||
|
ADD COLUMN rate_limit_time_window_seconds INTEGER;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change 8: Empty Sections Message
|
||||||
|
|
||||||
|
### What
|
||||||
|
When a synthesis has empty or null `sections`, show "Aucune section trouvee dans cette synthese." instead of rendering nothing.
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
- In `SynthesisDetail.tsx`: wrap sections rendering in `<Show when={sections.length > 0} fallback={...}>` with italic centered message
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
- In `models/synthesis.rs`: update `SynthesisResponse::try_from` to handle null/invalid sections gracefully (return empty vec instead of error)
|
||||||
|
- In `services/email.rs`: add fallback text in email template when sections is empty
|
||||||
|
|
||||||
|
### Files to modify
|
||||||
|
- `frontend/src/pages/SynthesisDetail.tsx` — empty state fallback
|
||||||
|
- `frontend/src/i18n/fr.ts` — add `'synthesis.noSections'` key
|
||||||
|
- `backend/src/models/synthesis.rs` — graceful null/empty sections handling
|
||||||
|
- `backend/src/services/email.rs` — email template empty state
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Order
|
||||||
|
|
||||||
|
Changes should be implemented in this order due to dependencies:
|
||||||
|
|
||||||
|
1. **Change 7** (migration) — adds all new columns
|
||||||
|
2. **Change 2** (provider/model selection) — depends on migration
|
||||||
|
3. **Change 3** (rate limit overrides) — depends on migration
|
||||||
|
4. **Change 1** (settings export/import) — depends on Change 2 for full field support
|
||||||
|
5. **Changes 4, 5, 6** (scraper/pipeline improvements) — independent of each other
|
||||||
|
6. **Change 8** (empty sections) — independent, can be done anytime
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verification Plan
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
```bash
|
||||||
|
cd backend && cargo check # compiles
|
||||||
|
cd backend && cargo test --lib # all unit tests pass
|
||||||
|
```
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
```bash
|
||||||
|
cd frontend && npx tsc --noEmit # type-checks
|
||||||
|
cd frontend && npx vitest run # all tests pass
|
||||||
|
cd frontend && npx vite build # production build succeeds
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manual Testing
|
||||||
|
1. Settings: select provider, pick research model, pick writing model, save, reload — values persisted
|
||||||
|
2. Settings: set rate limit overrides, save — verify generation respects them
|
||||||
|
3. Settings: export JSON (with/without keys), import JSON — form populated correctly
|
||||||
|
4. Generate: verify original titles preserved in output
|
||||||
|
5. Generate: verify homepage URLs filtered out
|
||||||
|
6. Generate: verify broken links with canonical /404 paths rejected
|
||||||
|
7. Generate: verify noindex pages rejected
|
||||||
|
8. View synthesis with empty sections: see fallback message
|
||||||
Loading…
Reference in New Issue