Add v2 changes implementation plan
10 tasks covering: migration, settings model expansion, DB queries, pipeline updates (user model selection, rate limiter, URL filter), prompts (deep-link, original title), scraper improvements, API key export, frontend (dual models, rate limits, export/import), and empty sections fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>master
parent
74e2cb0273
commit
6b27a0f691
@ -0,0 +1,927 @@
|
||||
# v2 Changes Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Implement 8 changes from the v2 functional spec — settings schema expansion, dual model selection, rate limit overrides, settings export/import, scraper improvements, deep-link enforcement, and empty sections fallback.
|
||||
|
||||
**Architecture:** All changes build on the existing Rust/Axum backend + SolidJS frontend + Postgres stack. The migration adds 5 columns to the `settings` table. Scraper improvements are isolated to `scraper.rs`. Pipeline changes touch `synthesis.rs` and `prompts.rs`. Frontend changes are primarily in `Settings.tsx` and `SynthesisDetail.tsx`.
|
||||
|
||||
**Tech Stack:** Rust (Axum, sqlx, scraper, reqwest), SolidJS, Postgres, Tailwind CSS v4
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-03-21-v2-changes-design.md`
|
||||
|
||||
**Spec-to-task traceability:**
|
||||
|
||||
| Spec Change | Plan Task(s) |
|
||||
|---|---|
|
||||
| Change 1: Settings Export/Import | Task 7 (backend), Task 8 (frontend) |
|
||||
| Change 2: Provider/Model + Two Dropdowns | Tasks 1-4, Task 8 |
|
||||
| Change 3: User Rate Limit Overrides | Tasks 1-3, Task 4 (pipeline), Task 8 |
|
||||
| Change 4: Original Title Preservation | Task 5 (prompts/model), Task 6 (scraper) |
|
||||
| Change 5: Enhanced Broken Link Detection | Task 6 (scraper) |
|
||||
| Change 6: Deep-Link URL Enforcement | Task 4 (pipeline), Task 5 (prompt) |
|
||||
| Change 7: Database Migration | Task 1 |
|
||||
| Change 8: Empty Sections Message | Task 5 (backend model), Task 9 (frontend + email) |
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
### Backend — New Files
|
||||
- `backend/migrations/20260321000010_add_settings_v2_fields.sql` — migration for 5 new columns
|
||||
|
||||
### Backend — Modified Files
|
||||
- `backend/src/models/settings.rs` — add 5 fields to structs + validation + max-length on new strings
|
||||
- `backend/src/db/settings.rs` — update SQL queries for new columns
|
||||
- `backend/src/handlers/settings.rs` — no code changes needed (auto-adapts via `From<UserSettings>` and `Json<UpdateSettingsRequest>`)
|
||||
- `backend/src/handlers/api_keys.rs` — add `export_keys` handler
|
||||
- `backend/src/router.rs` — add export route
|
||||
- `backend/src/services/scraper.rs` — enhanced title extraction, broken link detection (canonical, short-page, noindex, redirect URL)
|
||||
- `backend/src/services/prompts.rs` — deep-link instruction, original title in rewrite prompt
|
||||
- `backend/src/services/synthesis.rs` — user provider/model selection, per-generation rate limiter, homepage URL filter
|
||||
- `backend/src/models/synthesis.rs` — `original_title` on ScrapedNewsItem (with serde rename), null-safe SynthesisResponse, update breaking test
|
||||
- `backend/src/services/email.rs` — empty sections fallback in email template
|
||||
|
||||
### Frontend — Modified Files
|
||||
- `frontend/src/types.ts` — add `ai_model_writing`, optional rate limit fields
|
||||
- `frontend/src/pages/Settings.tsx` — dual model dropdowns, rate limit section, export/import
|
||||
- `frontend/src/pages/SynthesisDetail.tsx` — empty sections fallback
|
||||
- `frontend/src/api/apiKeys.ts` — add `exportKeys()`
|
||||
- `frontend/src/i18n/fr.ts` — new translation keys
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Database Migration
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/migrations/20260321000010_add_settings_v2_fields.sql`
|
||||
|
||||
- [ ] **Step 1: Write migration SQL**
|
||||
|
||||
```sql
|
||||
-- Add v2 settings fields: provider/model selection, writing model, rate limit overrides
|
||||
ALTER TABLE settings
|
||||
ADD COLUMN ai_provider VARCHAR(100) NOT NULL DEFAULT '',
|
||||
ADD COLUMN ai_model VARCHAR(100) NOT NULL DEFAULT '',
|
||||
ADD COLUMN ai_model_writing VARCHAR(100) NOT NULL DEFAULT '',
|
||||
ADD COLUMN rate_limit_max_requests INTEGER,
|
||||
ADD COLUMN rate_limit_time_window_seconds INTEGER;
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify backend compiles**
|
||||
|
||||
Run: `cd backend && cargo check`
|
||||
Expected: compiles (migration is SQL, checked at runtime)
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/migrations/20260321000010_add_settings_v2_fields.sql
|
||||
git commit -m "migration: add v2 settings fields (provider, models, rate limits)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Backend Settings Model Expansion
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/models/settings.rs`
|
||||
|
||||
- [ ] **Step 1: Write tests for new fields**
|
||||
|
||||
Add to the existing `#[cfg(test)] mod tests` in `settings.rs`:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_validate_with_model_fields() {
|
||||
let req = UpdateSettingsRequest {
|
||||
theme: "AI".into(),
|
||||
max_age_days: 7,
|
||||
categories: vec!["News".into()],
|
||||
max_items_per_category: 4,
|
||||
search_agent_behavior: "".into(),
|
||||
ai_provider: "gemini".into(),
|
||||
ai_model: "gemini-2.5-flash".into(),
|
||||
ai_model_writing: "gemini-2.5-pro".into(),
|
||||
rate_limit_max_requests: None,
|
||||
rate_limit_time_window_seconds: None,
|
||||
};
|
||||
assert!(req.validate().is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_validate_rate_limit_zero_rejected() {
|
||||
let mut req = valid_request();
|
||||
req.rate_limit_max_requests = Some(0);
|
||||
assert!(req.validate().is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_validate_rate_limit_zero_window_rejected() {
|
||||
let mut req = valid_request();
|
||||
req.rate_limit_time_window_seconds = Some(0);
|
||||
assert!(req.validate().is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_validate_rate_limit_valid() {
|
||||
let mut req = valid_request();
|
||||
req.rate_limit_max_requests = Some(10);
|
||||
req.rate_limit_time_window_seconds = Some(60);
|
||||
assert!(req.validate().is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_validate_ai_provider_too_long_rejected() {
|
||||
let mut req = valid_request();
|
||||
req.ai_provider = "a".repeat(101);
|
||||
assert!(req.validate().is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_validate_ai_model_too_long_rejected() {
|
||||
let mut req = valid_request();
|
||||
req.ai_model = "a".repeat(101);
|
||||
assert!(req.validate().is_err());
|
||||
}
|
||||
|
||||
fn valid_request() -> UpdateSettingsRequest {
|
||||
UpdateSettingsRequest {
|
||||
theme: "AI".into(),
|
||||
max_age_days: 7,
|
||||
categories: vec!["News".into()],
|
||||
max_items_per_category: 4,
|
||||
search_agent_behavior: "".into(),
|
||||
ai_provider: "".into(),
|
||||
ai_model: "".into(),
|
||||
ai_model_writing: "".into(),
|
||||
rate_limit_max_requests: None,
|
||||
rate_limit_time_window_seconds: None,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify compilation fails**
|
||||
|
||||
Run: `cd backend && cargo test --lib settings`
|
||||
Expected: COMPILATION FAILURE — new fields don't exist on struct yet
|
||||
|
||||
- [ ] **Step 3: Add fields to all settings structs**
|
||||
|
||||
In `UserSettings` (line 9), add after `search_agent_behavior`:
|
||||
```rust
|
||||
pub ai_provider: String,
|
||||
pub ai_model: String,
|
||||
pub ai_model_writing: String,
|
||||
pub rate_limit_max_requests: Option<i32>,
|
||||
pub rate_limit_time_window_seconds: Option<i32>,
|
||||
```
|
||||
|
||||
In `SettingsResponse` (line 20), add the same 5 fields. Update the `From<UserSettings>` impl (line 29) to map them.
|
||||
|
||||
In `UpdateSettingsRequest` (line 42), add the same 5 fields. Update `validate()` (line 51), add after existing checks:
|
||||
```rust
|
||||
// Validate string length for new fields
|
||||
if self.ai_provider.len() > 100 {
|
||||
return Err("ai_provider must be at most 100 characters".into());
|
||||
}
|
||||
if self.ai_model.len() > 100 {
|
||||
return Err("ai_model must be at most 100 characters".into());
|
||||
}
|
||||
if self.ai_model_writing.len() > 100 {
|
||||
return Err("ai_model_writing must be at most 100 characters".into());
|
||||
}
|
||||
// Validate rate limit overrides
|
||||
if let Some(max_req) = self.rate_limit_max_requests {
|
||||
if max_req < 1 {
|
||||
return Err("rate_limit_max_requests must be at least 1".into());
|
||||
}
|
||||
}
|
||||
if let Some(window) = self.rate_limit_time_window_seconds {
|
||||
if window < 1 {
|
||||
return Err("rate_limit_time_window_seconds must be at least 1".into());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
In `default_settings()` (line 94), add: `ai_provider: String::new()`, `ai_model: String::new()`, `ai_model_writing: String::new()`, `rate_limit_max_requests: None`, `rate_limit_time_window_seconds: None`.
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `cd backend && cargo test --lib settings`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/models/settings.rs
|
||||
git commit -m "feat: add v2 fields to settings model (provider, models, rate limits)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Backend Settings DB Queries
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/db/settings.rs`
|
||||
|
||||
- [ ] **Step 1: Update SettingsRow struct**
|
||||
|
||||
Add to `SettingsRow` (line 13):
|
||||
```rust
|
||||
pub ai_provider: String,
|
||||
pub ai_model: String,
|
||||
pub ai_model_writing: String,
|
||||
pub rate_limit_max_requests: Option<i32>,
|
||||
pub rate_limit_time_window_seconds: Option<i32>,
|
||||
```
|
||||
|
||||
Update `TryFrom<SettingsRow> for UserSettings` (line 24) to map the new fields.
|
||||
|
||||
- [ ] **Step 2: Update get_or_create_default query**
|
||||
|
||||
In `get_or_create_default()` (line 48), add new columns to INSERT defaults and SELECT. The INSERT uses defaults from the table definition (empty strings, NULLs).
|
||||
|
||||
- [ ] **Step 3: Update upsert query**
|
||||
|
||||
In `upsert()` (line 78), add new columns to the UPDATE SET clause and bind the new parameters.
|
||||
|
||||
- [ ] **Step 4: Verify compilation and run tests**
|
||||
|
||||
Run: `cd backend && cargo check && cargo test --lib`
|
||||
Expected: compiles clean, all tests pass
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/db/settings.rs
|
||||
git commit -m "feat: update settings DB queries for v2 fields"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Backend Pipeline — User Provider/Model Selection, Rate Limiter, URL Filter
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/services/synthesis.rs`
|
||||
|
||||
- [ ] **Step 1: Write tests**
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_homepage_url_filtered() {
|
||||
let url = url::Url::parse("https://example.com/").unwrap();
|
||||
assert!(url.path() == "/" || url.path().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_article_url_not_filtered() {
|
||||
let url = url::Url::parse("https://example.com/news/article-123").unwrap();
|
||||
assert!(url.path() != "/" && !url.path().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_homepage_without_trailing_slash_filtered() {
|
||||
// url::Url normalizes "https://example.com" to path "/"
|
||||
let url = url::Url::parse("https://example.com").unwrap();
|
||||
assert_eq!(url.path(), "/");
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Update resolve_provider_and_key() to respect user's ai_provider**
|
||||
|
||||
In `resolve_provider_and_key()` (line 374), change the logic:
|
||||
- Accept `&UserSettings` as a new parameter
|
||||
- If `settings.ai_provider` is non-empty, look up the user's key for that specific provider
|
||||
- If no key found for the preferred provider, return an error telling the user to configure their key
|
||||
- If `settings.ai_provider` is empty, fall back to existing behavior (first available key)
|
||||
|
||||
Updated signature:
|
||||
```rust
|
||||
async fn resolve_provider_and_key(
|
||||
state: &AppState,
|
||||
user_id: Uuid,
|
||||
settings: &UserSettings,
|
||||
) -> Result<(String, String), AppError>
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Update model resolution to use user settings**
|
||||
|
||||
After `resolve_provider_and_key()`, determine models:
|
||||
```rust
|
||||
let model_research = if !settings.ai_model.is_empty() {
|
||||
settings.ai_model.clone()
|
||||
} else {
|
||||
resolve_model(state, &provider_name).await?
|
||||
};
|
||||
|
||||
let model_writing = if !settings.ai_model_writing.is_empty() {
|
||||
settings.ai_model_writing.clone()
|
||||
} else {
|
||||
model_research.clone() // fall back to research model
|
||||
};
|
||||
```
|
||||
|
||||
Pass `model_research` to `generate_search_pass()` and `model_writing` to `generate_rewrite_pass()`.
|
||||
|
||||
- [ ] **Step 4: Add per-generation rate limiter**
|
||||
|
||||
Before Pass 1 and Pass 2 rate limit checks:
|
||||
```rust
|
||||
// Use user's rate limit overrides if set, otherwise use global provider limiter
|
||||
match (settings.rate_limit_max_requests, settings.rate_limit_time_window_seconds) {
|
||||
(Some(max_req), Some(window_sec)) => {
|
||||
let temp_limiter = RateLimiter::new(max_req as u32, (window_sec as u64) * 1000);
|
||||
temp_limiter.acquire().await;
|
||||
}
|
||||
_ => {
|
||||
state.provider_rate_limiter.acquire(&provider_name).await?;
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add homepage URL filter after Pass 1**
|
||||
|
||||
After `parse_llm_output()`, before scraping. Add `use url::Url;` at top of file:
|
||||
```rust
|
||||
// Filter out homepage/root URLs
|
||||
for (_category, items) in parsed_items.iter_mut() {
|
||||
let before = items.len();
|
||||
items.retain(|item| {
|
||||
match Url::parse(&item.url) {
|
||||
Ok(parsed) => parsed.path() != "/" && !parsed.path().is_empty(),
|
||||
Err(_) => false,
|
||||
}
|
||||
});
|
||||
let filtered = before - items.len();
|
||||
if filtered > 0 {
|
||||
tracing::warn!(count = filtered, "Filtered homepage/unparseable URLs");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run tests**
|
||||
|
||||
Run: `cd backend && cargo test --lib synthesis`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/services/synthesis.rs
|
||||
git commit -m "feat: user provider/model selection, per-generation rate limiter, homepage URL filter"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Backend — Prompts, Original Title, Null-Safe Sections
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/services/prompts.rs`
|
||||
- Modify: `backend/src/models/synthesis.rs`
|
||||
|
||||
- [ ] **Step 1: Add original_title to ScrapedNewsItem with serde rename**
|
||||
|
||||
In `models/synthesis.rs` (line 144), add to `ScrapedNewsItem`:
|
||||
```rust
|
||||
#[serde(rename = "originalTitle")]
|
||||
pub original_title: String,
|
||||
```
|
||||
|
||||
The `rename` matches the camelCase convention used by `scrapedContent` and ensures the rewrite prompt JSON uses `"originalTitle"` as referenced in the prompt text.
|
||||
|
||||
- [ ] **Step 2: Handle null sections in SynthesisResponse**
|
||||
|
||||
Replace `TryFrom<Synthesis> for SynthesisResponse` (line 47-67):
|
||||
```rust
|
||||
impl TryFrom<Synthesis> for SynthesisResponse {
|
||||
type Error = crate::errors::AppError;
|
||||
|
||||
fn try_from(s: Synthesis) -> Result<Self, Self::Error> {
|
||||
let sections: Vec<NewsSection> = if s.sections.is_null() {
|
||||
Vec::new()
|
||||
} else {
|
||||
serde_json::from_value(s.sections).unwrap_or_default()
|
||||
};
|
||||
|
||||
Ok(Self {
|
||||
id: s.id,
|
||||
week: s.week,
|
||||
sections,
|
||||
status: s.status,
|
||||
created_at: s.created_at,
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Update or remove the breaking test**
|
||||
|
||||
The existing test `synthesis_response_from_invalid_json_fails` (if it exists) asserts that invalid JSON causes an error. Since we now use `unwrap_or_default()`, update this test to assert that invalid JSON produces an empty sections vec instead:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_synthesis_response_from_null_sections_returns_empty() {
|
||||
let s = Synthesis {
|
||||
id: Uuid::new_v4(),
|
||||
user_id: Uuid::new_v4(),
|
||||
week: "2026-W12".into(),
|
||||
sections: serde_json::Value::Null,
|
||||
status: "completed".into(),
|
||||
created_at: Utc::now(),
|
||||
};
|
||||
let response = SynthesisResponse::try_from(s).unwrap();
|
||||
assert!(response.sections.is_empty());
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Update build_search_prompt with deep-link instruction**
|
||||
|
||||
In `prompts.rs`, add to the search prompt user text (around line 70):
|
||||
```
|
||||
"Ne retourne JAMAIS des URLs de pages d'accueil (homepage). Fournis toujours des liens directs vers des articles specifiques avec un chemin complet (pas juste le nom de domaine)."
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Update build_rewrite_prompt with original title + language rules**
|
||||
|
||||
Update `build_rewrite_prompt()` (line 99). The `ScrapedNewsItem` data now includes `originalTitle` in the serialized JSON. Add to the prompt instructions:
|
||||
```
|
||||
"Pour chaque article, un 'originalTitle' extrait de la page web est fourni. Utilise ce titre original comme base pour le titre final.
|
||||
Regles linguistiques :
|
||||
- Les titres en anglais doivent rester en anglais (ne pas traduire).
|
||||
- Les titres en francais doivent rester en francais.
|
||||
- Les titres dans d'autres langues doivent etre traduits en francais."
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Update existing prompt tests**
|
||||
|
||||
Update any prompt tests that check exact string content to include the new instructions. Update any tests constructing `ScrapedNewsItem` to include `original_title`.
|
||||
|
||||
- [ ] **Step 7: Run tests**
|
||||
|
||||
Run: `cd backend && cargo test --lib`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/services/prompts.rs backend/src/models/synthesis.rs
|
||||
git commit -m "feat: deep-link enforcement, original title preservation, null-safe sections"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Backend Scraper — Enhanced Title + Broken Link Detection
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/services/scraper.rs`
|
||||
|
||||
- [ ] **Step 1: Write tests for enhanced title extraction**
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_title_priority_title_element_first() {
|
||||
let html = r#"<html><head><title>From Title</title><meta property="og:title" content="From OG"></head><body><h1>From H1</h1></body></html>"#;
|
||||
let doc = Html::parse_document(html);
|
||||
assert_eq!(extract_page_title(&doc), Some("From Title".into()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_title_fallback_to_og_title() {
|
||||
let html = r#"<html><head><title></title><meta property="og:title" content="From OG"></head><body></body></html>"#;
|
||||
let doc = Html::parse_document(html);
|
||||
assert_eq!(extract_page_title(&doc), Some("From OG".into()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_title_fallback_to_h1() {
|
||||
let html = r#"<html><head></head><body><h1>From H1</h1></body></html>"#;
|
||||
let doc = Html::parse_document(html);
|
||||
assert_eq!(extract_page_title(&doc), Some("From H1".into()));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write tests for canonical/redirect error detection**
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_canonical_404_detected() {
|
||||
let html = r#"<html><head><link rel="canonical" href="https://example.com/404"></head><body><p>Not found</p></body></html>"#;
|
||||
let doc = Html::parse_document(html);
|
||||
assert!(detect_canonical_error(&doc));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_og_url_error_path_detected() {
|
||||
let html = r#"<html><head><meta property="og:url" content="https://example.com/error"></head><body></body></html>"#;
|
||||
let doc = Html::parse_document(html);
|
||||
assert!(detect_canonical_error(&doc));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_canonical_normal_url_not_flagged() {
|
||||
let html = r#"<html><head><link rel="canonical" href="https://example.com/news/article-123"></head><body></body></html>"#;
|
||||
let doc = Html::parse_document(html);
|
||||
assert!(!detect_canonical_error(&doc));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Write tests for short-page body text detection**
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_short_page_with_error_phrases_detected() {
|
||||
assert!(detect_short_page_error("Sorry, this page could not be found. Please try again."));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_short_page_french_error_detected() {
|
||||
assert!(detect_short_page_error("Desolee, cette page introuvable."));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_long_page_not_flagged() {
|
||||
let long_text = "word ".repeat(400); // > 1500 chars
|
||||
assert!(!detect_short_page_error(&long_text));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_short_page_404_near_error() {
|
||||
assert!(detect_short_page_error("Error 404 - the page was not found on this server."));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Write tests for noindex and redirect URL detection**
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_noindex_detected() {
|
||||
let html = r#"<html><head><meta name="robots" content="noindex"></head><body><p>Content</p></body></html>"#;
|
||||
let doc = Html::parse_document(html);
|
||||
assert!(detect_noindex(&doc));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_noindex_not_present() {
|
||||
let html = r#"<html><head><meta name="robots" content="index,follow"></head><body></body></html>"#;
|
||||
let doc = Html::parse_document(html);
|
||||
assert!(!detect_noindex(&doc));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_redirect_url_error_path() {
|
||||
assert!(is_error_path("/404"));
|
||||
assert!(is_error_path("/404.html"));
|
||||
assert!(is_error_path("/error"));
|
||||
assert!(is_error_path("/not-found"));
|
||||
assert!(!is_error_path("/news/article-123"));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run tests to verify they fail**
|
||||
|
||||
Run: `cd backend && cargo test --lib scraper`
|
||||
Expected: COMPILATION FAILURE — new functions don't exist yet
|
||||
|
||||
- [ ] **Step 6: Implement enhanced extract_page_title()**
|
||||
|
||||
Replace existing function (line 257). Priority: `<title>` -> `og:title` -> `<h1>` -> None:
|
||||
|
||||
```rust
|
||||
fn extract_page_title(doc: &Html) -> Option<String> {
|
||||
// 1. <title> element
|
||||
if let Ok(sel) = Selector::parse("title") {
|
||||
if let Some(el) = doc.select(&sel).next() {
|
||||
let text = el.text().collect::<String>().trim().to_string();
|
||||
if !text.is_empty() {
|
||||
return Some(text);
|
||||
}
|
||||
}
|
||||
}
|
||||
// 2. og:title meta tag
|
||||
if let Ok(sel) = Selector::parse(r#"meta[property="og:title"]"#) {
|
||||
if let Some(el) = doc.select(&sel).next() {
|
||||
if let Some(content) = el.value().attr("content") {
|
||||
let text = content.trim().to_string();
|
||||
if !text.is_empty() {
|
||||
return Some(text);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// 3. First <h1>
|
||||
if let Ok(sel) = Selector::parse("h1") {
|
||||
if let Some(el) = doc.select(&sel).next() {
|
||||
let text = el.text().collect::<String>().trim().to_string();
|
||||
if !text.is_empty() {
|
||||
return Some(text);
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Implement detect_canonical_error(), detect_short_page_error(), detect_noindex(), is_error_path()**
|
||||
|
||||
(Code as shown in previous version — `detect_canonical_error` checks canonical+og:url, `detect_short_page_error` checks body < 1500 chars for error phrases, `detect_noindex` checks robots meta, `is_error_path` checks path against known error patterns)
|
||||
|
||||
- [ ] **Step 8: Integrate into scrape_url()**
|
||||
|
||||
In `scrape_url()` (line 72), after `let response = ...`:
|
||||
1. **Capture final URL before consuming body:** `let final_url = response.url().clone();`
|
||||
2. After reading `bytes` and parsing `document`:
|
||||
3. Check `is_error_path(final_url.path())` — if true, set `is_soft_404 = true`
|
||||
4. Check `detect_canonical_error(&document)` — if true, set `is_soft_404 = true`
|
||||
5. Check `detect_noindex(&document)` — if true, set `is_soft_404 = true`
|
||||
6. After extracting `body_text` (existing `extract_body_text` call), check `detect_short_page_error(&body_text)` — if true, set `is_soft_404 = true`
|
||||
|
||||
Combine with existing `detect_soft_404()` using logical OR:
|
||||
```rust
|
||||
let is_soft_404 = detect_soft_404(&document)
|
||||
|| detect_canonical_error(&document)
|
||||
|| detect_noindex(&document)
|
||||
|| detect_short_page_error(&body_text)
|
||||
|| is_error_path(final_url.path());
|
||||
```
|
||||
|
||||
- [ ] **Step 9: Run tests**
|
||||
|
||||
Run: `cd backend && cargo test --lib scraper`
|
||||
Expected: ALL PASS
|
||||
|
||||
- [ ] **Step 10: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/services/scraper.rs
|
||||
git commit -m "feat: enhanced title extraction, broken link detection (canonical, short-page, noindex, redirect)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Backend API Key Export Endpoint
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/handlers/api_keys.rs`
|
||||
- Modify: `backend/src/router.rs`
|
||||
|
||||
- [ ] **Step 1: Add export_keys handler**
|
||||
|
||||
In `api_keys.rs`:
|
||||
```rust
|
||||
/// `POST /api/v1/user/api-keys/export`
|
||||
///
|
||||
/// Returns decrypted API keys for settings export. Rate-limited, audit-logged.
|
||||
/// Uses POST (not GET) to require CSRF token.
|
||||
pub async fn export_keys(
|
||||
auth_user: AuthUser,
|
||||
State(state): State<AppState>,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
if !state.auth_rate_limiter.check(&format!("key-export:{}", auth_user.id)) {
|
||||
return Err(AppError::RateLimited("Too many export requests".into()));
|
||||
}
|
||||
|
||||
let stored_keys = db::api_keys::list_for_user(&state.pool, auth_user.id).await?;
|
||||
let master_key = encryption::MasterKey::from_hex(&state.config.master_encryption_key)?;
|
||||
|
||||
let mut exported = Vec::new();
|
||||
for key in &stored_keys {
|
||||
let decrypted = encryption::decrypt(&master_key, &key.encrypted_key, &key.nonce)?;
|
||||
exported.push(serde_json::json!({
|
||||
"provider_name": key.provider_name,
|
||||
"api_key": decrypted,
|
||||
}));
|
||||
}
|
||||
|
||||
tracing::info!(user_id = %auth_user.id, key_count = exported.len(), "API keys exported");
|
||||
Ok(Json(exported))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add route**
|
||||
|
||||
In `router.rs`, add after the existing api-keys routes:
|
||||
```rust
|
||||
.route("/user/api-keys/export", post(handlers::api_keys::export_keys))
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify compilation**
|
||||
|
||||
Run: `cd backend && cargo check`
|
||||
Expected: compiles
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/handlers/api_keys.rs backend/src/router.rs
|
||||
git commit -m "feat: API key export endpoint for settings backup"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Frontend — Dual Model Dropdowns + Rate Limits + Export/Import
|
||||
|
||||
**Files:**
|
||||
- Modify: `frontend/src/types.ts`
|
||||
- Modify: `frontend/src/pages/Settings.tsx`
|
||||
- Modify: `frontend/src/api/apiKeys.ts`
|
||||
- Modify: `frontend/src/i18n/fr.ts`
|
||||
|
||||
- [ ] **Step 1: Update TypeScript types**
|
||||
|
||||
In `types.ts`, update `UserSettings`:
|
||||
```typescript
|
||||
export interface UserSettings {
|
||||
theme: string;
|
||||
max_age_days: number;
|
||||
max_items_per_category: number;
|
||||
search_agent_behavior: string;
|
||||
ai_provider: string;
|
||||
ai_model: string;
|
||||
ai_model_writing: string;
|
||||
categories: string[];
|
||||
rate_limit_max_requests: number | null;
|
||||
rate_limit_time_window_seconds: number | null;
|
||||
}
|
||||
```
|
||||
|
||||
Update `DEFAULT_SETTINGS` to include `ai_model_writing: ''`, `rate_limit_max_requests: null`, `rate_limit_time_window_seconds: null`.
|
||||
|
||||
- [ ] **Step 2: Add i18n keys**
|
||||
|
||||
Add to `fr.ts`:
|
||||
```typescript
|
||||
'settings.modelResearch': "Modele d'IA (Recherche et Extraction)",
|
||||
'settings.modelResearchHelp': "Choisissez le modele d'IA utilise pour rechercher et extraire les informations.",
|
||||
'settings.modelWriting': "Modele d'IA (Redaction et Synthese)",
|
||||
'settings.modelWritingHelp': "Choisissez le modele d'IA utilise pour le second agent, charge de rediger et structurer la synthese finale.",
|
||||
'settings.rateLimitSection': 'Limitation de taux',
|
||||
'settings.rateLimitMaxRequests': 'Requetes maximum',
|
||||
'settings.rateLimitTimeWindow': 'Fenetre de temps (secondes)',
|
||||
'settings.rateLimitHelp': "Configurez le nombre maximum de requetes autorisees pendant la fenetre de temps specifiee. Laissez vide pour utiliser les valeurs par defaut de l'administrateur.",
|
||||
'settings.rateLimitEffective': '{max} requetes / {window} secondes',
|
||||
'settings.rateLimitReset': 'Reinitialiser',
|
||||
'settings.export': 'Exporter',
|
||||
'settings.import': 'Importer',
|
||||
'settings.exportIncludeKeys': 'Inclure les cles API',
|
||||
'settings.exportKeysWarning': 'Les cles API seront incluses en clair dans le fichier. Ne partagez pas ce fichier.',
|
||||
'settings.importSuccess': "Configuration importee avec succes. N'oubliez pas d'enregistrer.",
|
||||
'settings.importError': "Erreur lors de l'importation du fichier JSON.",
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add exportKeys to API client**
|
||||
|
||||
In `api/apiKeys.ts`:
|
||||
```typescript
|
||||
exportKeys: (): Promise<{ provider_name: string; api_key: string }[]> =>
|
||||
api.post('/user/api-keys/export'),
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Update Settings page — dual model dropdowns**
|
||||
|
||||
Replace the single model dropdown with two:
|
||||
- First: label `t('settings.modelResearch')`, bound to `settings().ai_model`, help text `t('settings.modelResearchHelp')`
|
||||
- Second: label `t('settings.modelWriting')`, bound to `settings().ai_model_writing`, help text `t('settings.modelWritingHelp')`
|
||||
- Both filtered by the selected provider's models
|
||||
|
||||
- [ ] **Step 5: Add rate limit section to Settings page**
|
||||
|
||||
Below the categories section, add a horizontal rule separator and rate limit section:
|
||||
- Two number inputs: max_requests (min: 1) and time_window_seconds (min: 1)
|
||||
- Both use `number | null` — empty input maps to null
|
||||
- Help text explaining admin defaults are used when empty
|
||||
- Effective rate display when both values are set
|
||||
- "Reinitialiser" link that sets both to null
|
||||
|
||||
- [ ] **Step 6: Add export/import to Settings page header**
|
||||
|
||||
Add Download and Upload icon buttons next to the title. On export:
|
||||
- Fetch current settings from form state
|
||||
- If "Inclure les cles API" checkbox is checked, also call `apiKeysApi.exportKeys()`
|
||||
- Merge into JSON, trigger download as `settings.json`
|
||||
|
||||
On import:
|
||||
- File picker for `.json` files only
|
||||
- Parse uploaded JSON, validate basic shape
|
||||
- Populate form fields with imported values (merge over defaults for missing fields)
|
||||
- If API keys present in the import, call `apiKeysApi.create()` for each key
|
||||
- Show success banner: "N'oubliez pas d'enregistrer"
|
||||
- Invalid JSON shows error banner
|
||||
|
||||
- [ ] **Step 7: Update settings-validation tests for new fields**
|
||||
|
||||
Update existing tests in `__tests__/settings-validation.test.ts` to include the new fields in `DEFAULT_SETTINGS` assertions.
|
||||
|
||||
- [ ] **Step 8: Verify TypeScript and run tests**
|
||||
|
||||
Run: `cd frontend && npx tsc --noEmit && npx vitest run`
|
||||
Expected: no errors, all tests pass
|
||||
|
||||
- [ ] **Step 9: Commit**
|
||||
|
||||
```bash
|
||||
git add frontend/src/types.ts frontend/src/pages/Settings.tsx frontend/src/api/apiKeys.ts frontend/src/i18n/fr.ts frontend/src/__tests__/settings-validation.test.ts
|
||||
git commit -m "feat: dual model selection, rate limit overrides, settings export/import"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 9: Frontend + Backend — Empty Sections Fallback
|
||||
|
||||
**Files:**
|
||||
- Modify: `frontend/src/pages/SynthesisDetail.tsx`
|
||||
- Modify: `frontend/src/i18n/fr.ts`
|
||||
- Modify: `backend/src/services/email.rs`
|
||||
|
||||
- [ ] **Step 1: Add i18n key**
|
||||
|
||||
```typescript
|
||||
'synthesis.noSections': 'Aucune section trouvee dans cette synthese.',
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add fallback in SynthesisDetail**
|
||||
|
||||
Wrap the sections `<For>` loop (line 357) in a `<Show>`:
|
||||
```tsx
|
||||
<Show
|
||||
when={synth().sections && synth().sections.length > 0}
|
||||
fallback={
|
||||
<p class="text-center text-gray-500 italic py-12">
|
||||
{t('synthesis.noSections')}
|
||||
</p>
|
||||
}
|
||||
>
|
||||
<For each={synth().sections}>
|
||||
{(section) => <Section title={section.title} items={section.items} />}
|
||||
</For>
|
||||
</Show>
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit frontend changes**
|
||||
|
||||
```bash
|
||||
git add frontend/src/pages/SynthesisDetail.tsx frontend/src/i18n/fr.ts
|
||||
git commit -m "feat: empty sections fallback message in synthesis detail view"
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add fallback in email template + test**
|
||||
|
||||
In `backend/src/services/email.rs`, in `build_synthesis_html()`, add before iterating sections:
|
||||
```rust
|
||||
if sections.is_empty() {
|
||||
html.push_str("<p style=\"text-align:center;color:#6b7280;font-style:italic;padding:24px 0;\">Aucune section trouvee dans cette synthese.</p>");
|
||||
} else {
|
||||
// existing section rendering loop
|
||||
}
|
||||
```
|
||||
|
||||
Same for `build_synthesis_text()`.
|
||||
|
||||
Add test:
|
||||
```rust
|
||||
#[test]
|
||||
fn test_synthesis_html_empty_sections_shows_fallback() {
|
||||
let html = build_synthesis_html("2026-W12", "21 mars 2026", &[]);
|
||||
assert!(html.contains("Aucune section trouvee"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_synthesis_text_empty_sections_shows_fallback() {
|
||||
let text = build_synthesis_text("2026-W12", "21 mars 2026", &[]);
|
||||
assert!(text.contains("Aucune section trouvee"));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run backend tests**
|
||||
|
||||
Run: `cd backend && cargo test --lib email`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 6: Commit backend changes**
|
||||
|
||||
```bash
|
||||
git add backend/src/services/email.rs
|
||||
git commit -m "feat: empty sections fallback in email template"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 10: Final Verification
|
||||
|
||||
- [ ] **Step 1: Full backend check**
|
||||
|
||||
Run: `cd backend && cargo check && cargo test --lib`
|
||||
Expected: compiles, all tests pass
|
||||
|
||||
- [ ] **Step 2: Full frontend check**
|
||||
|
||||
Run: `cd frontend && npx tsc --noEmit && npx vitest run && npx vite build`
|
||||
Expected: type-checks, tests pass, builds
|
||||
|
||||
- [ ] **Step 3: Final commit if any uncommitted changes**
|
||||
|
||||
```bash
|
||||
git add -A && git status
|
||||
# Only commit if there are changes
|
||||
git commit -m "v2 changes complete: dual models, rate limits, export/import, scraper improvements"
|
||||
```
|
||||
Loading…
Reference in New Issue