docs: add spec for Autre fill-up to 75% synthesis target

master
oabrivard 3 months ago
parent cea723f7d7
commit aebd436a91

@ -0,0 +1,95 @@
# Design: "Autre" Fill-Up to Reach 75% Synthesis Target
**Date**: 2026-03-24
**Scope**: When total article count is low, expand "Autre" category to bring synthesis to 75% of maximum capacity
---
## Context
After the two-phase pipeline (personalized sources + web search), article drops (history filtering, validation failures, scrape errors) can leave syntheses with few articles. The "Autre" category currently caps at `max_items_per_category` like other categories, meaning overflow articles are silently dropped even when the synthesis is under-filled.
## Approach
After both phases complete, check if the total article count is below 75% of the maximum. If so, expand "Autre" capacity and fill from overflow articles that were dropped during classification.
## Target Calculation
- Maximum articles = `categories.len() × max_items_per_category` (user categories only)
- Target = `(0.75 × maximum).ceil()` as `usize`
- Shortfall = `target.saturating_sub(current_total)` (saturating to avoid panic if total exceeds target)
- If shortfall > 0, add overflow articles to "Autre" up to the shortfall
**Why exclude "Autre" from the maximum:** The goal is to ensure user-defined categories are adequately filled. "Autre" is the overflow bucket — it should not inflate the target. If user categories are well-filled, no fill-up is needed even if "Autre" is empty.
Example: 4 categories × 4 items = 16 max. Target = 12. If user categories have 8 articles and "Autre" has 2 (total 10), shortfall = 2. "Autre" accepts 2 more overflow articles.
## Mechanism
### 1. Collect overflow during classification
Modify `parse_classification_response` to return a second value: `Vec<ScrapedNewsItem>` of overflow articles — articles that were dropped because both their target category AND "Autre" were full.
Current signature:
```rust
fn parse_classification_response(...) -> HashMap<String, Vec<ScrapedNewsItem>>
```
New signature:
```rust
fn parse_classification_response(...) -> (HashMap<String, Vec<ScrapedNewsItem>>, Vec<ScrapedNewsItem>)
```
### 2. Accumulate overflow across phases
In `run_generation_inner`, collect overflow from both Phase 1 and Phase 2 classification calls into a single `all_overflow: Vec<ScrapedNewsItem>`.
### 3. Post-classification fill-up
After both phases, before the rewrite pass:
```
const SYNTHESIS_MIN_FILL_RATIO: f64 = 0.75;
total = sum of all articles in all_scraped
max_articles = categories.len() * max_items_per_category
target = (SYNTHESIS_MIN_FILL_RATIO * max_articles as f64).ceil() as usize
shortfall = target.saturating_sub(total)
if shortfall > 0 and all_overflow is non-empty:
// Filter overflow against max_articles_per_source (source diversity)
for each overflow article:
count domain occurrences in all_scraped
skip if domain already at max_articles_per_source
take up to shortfall valid overflow articles
add them to all_scraped["category_autre"]
```
**Source diversity enforcement:** Overflow articles added back to "Autre" must respect the `max_articles_per_source` limit. Count existing domain occurrences across all categories in `all_scraped`, and only add an overflow article if its domain is still under the limit.
## Hardcoded 75%
The 75% target is hardcoded as a constant `SYNTHESIS_MIN_FILL_RATIO: f64 = 0.75` in `synthesis.rs`. No user setting needed — this is an internal quality threshold.
## Best-effort
If there aren't enough overflow articles to reach 75%, the synthesis proceeds with whatever it has. No error is raised.
## Files to Modify
- **Modify:** `backend/src/services/synthesis.rs`:
- Add `SYNTHESIS_MIN_FILL_RATIO` constant
- Modify `parse_classification_response` signature and body to collect and return overflow
- Update 2 production call sites to destructure the tuple
- Update 5 existing classification unit tests for new return type
- Add fill-up logic in `run_generation_inner` between Phase 2 and rewrite pass
- Add unit tests for overflow collection and fill-up calculation
## What Does NOT Change
- `max_items_per_category` — still the cap for user categories
- Classification prompt — unchanged
- Rewrite pass — sees the final article set including expanded "Autre"
- Frontend — no changes
- Database — no changes
- No new settings
Loading…
Cancel
Save