ai_synth/docs/superpowers/plans/2026-03-24-autre-fillup.md

"Autre" Fill-Up — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: When total synthesis articles are below 75% capacity, expand "Autre" with overflow articles from classification.

Architecture: Modify parse_classification_response to return overflow. Add fill-up logic between Phase 2 and rewrite pass. Single-file change.

Tech Stack: Rust

Spec: docs/superpowers/specs/2026-03-24-autre-fillup-design.md

---

Task 1: Modify parse_classification_response to collect overflow

Files:

• Modify: backend/src/services/synthesis.rs

• Step 1: Add constant

At the top of the helper functions section, add:

/// Minimum fill ratio for synthesis. If total articles are below this percentage
/// of the maximum capacity, overflow articles are added to "Autre" to compensate.
const SYNTHESIS_MIN_FILL_RATIO: f64 = 0.75;

• Step 2: Change parse_classification_response return type and body

Change return type from HashMap<String, Vec<ScrapedNewsItem>> to (HashMap<String, Vec<ScrapedNewsItem>>, Vec<ScrapedNewsItem>).

Add let mut overflow: Vec<ScrapedNewsItem> = Vec::new(); at the start of the function.

At the two drop points where articles are silently discarded, push to overflow instead:

Drop point 1 (line ~1735-1743) — category full AND Autre full:

        if filled >= max {
            let autre_filled = filled_counts.get("Autre").copied().unwrap_or(0);
            if autre_filled < max {
                result.entry("category_autre".to_string()).or_default().push(articles[index].clone());
                *filled_counts.entry("Autre".to_string()).or_insert(0) += 1;
                assigned_indices.insert(index);
            } else {
                // Both category and Autre full — collect as overflow
                overflow.push(articles[index].clone());
                assigned_indices.insert(index);
            }
            continue;
        }

Drop point 2 (line ~1752-1759) — unclassified, Autre full:

    for (i, article) in articles.iter().enumerate() {
        if !assigned_indices.contains(&i) {
            let autre_filled = filled_counts.get("Autre").copied().unwrap_or(0);
            if autre_filled < max {
                result.entry("category_autre".to_string()).or_default().push(article.clone());
                *filled_counts.entry("Autre".to_string()).or_insert(0) += 1;
            } else {
                // Autre full — collect as overflow
                overflow.push(article.clone());
            }
        }
    }

Change the return from result to (result, overflow).

• Step 3: Update 2 production call sites

Phase 1 (line ~494):

// Before:
let phase1_classified = parse_classification_response(...);

// After:
let (phase1_classified, phase1_overflow) = parse_classification_response(...);

Phase 2 (line ~701):

// Before:
let phase2_classified = parse_classification_response(...);

// After:
let (phase2_classified, phase2_overflow) = parse_classification_response(...);

• Step 4: Update 5 existing tests

All 5 tests that call parse_classification_response need to destructure the tuple. Change each from:

let result = parse_classification_response(...);

To:

let (result, _overflow) = parse_classification_response(...);

For the test classification_respects_max_per_category, also verify overflow is captured:

let (result, overflow) = parse_classification_response(&response, &articles, &categories, 2, &mut filled);
assert_eq!(result.get("category_0").map(|v| v.len()), Some(2));
assert!(result.get("category_autre").map(|v| v.len()).unwrap_or(0) > 0);
// Articles that overflowed both AI News AND Autre should be in overflow
assert!(!overflow.is_empty() || filled.get("Autre").copied().unwrap_or(0) <= 2);

• Step 5: Run tests

Run: cd backend && cargo test --lib Expected: all tests pass

• Step 6: Commit

git add backend/src/services/synthesis.rs
git commit -m "feat: parse_classification_response collects overflow articles"

---

Task 2: Add fill-up logic and accumulate overflow in pipeline

Files:

• Modify: backend/src/services/synthesis.rs

• Step 1: Accumulate overflow in run_generation_inner

Add let mut all_overflow: Vec<ScrapedNewsItem> = Vec::new(); near the other all_scraped initialization (around line 305).

After Phase 1 classification (where phase1_overflow is captured), add:

all_overflow.extend(phase1_overflow);

After Phase 2 classification (where phase2_overflow is captured), add:

all_overflow.extend(phase2_overflow);

• Step 2: Add fill-up logic before rewrite pass

Between the "COMBINED REWRITE PASS" header (line ~719) and the empty-check (line ~722), insert:

    // Fill-up: if total articles are below 75% of max, expand "Autre" with overflow
    let total_articles: usize = all_scraped.values().map(|v| v.len()).sum();
    let max_articles = settings.categories.len() * settings.max_items_per_category as usize;
    let target = (SYNTHESIS_MIN_FILL_RATIO * max_articles as f64).ceil() as usize;
    let shortfall = target.saturating_sub(total_articles);

    if shortfall > 0 && !all_overflow.is_empty() {
        tracing::info!(
            total = total_articles,
            target = target,
            shortfall = shortfall,
            overflow_available = all_overflow.len(),
            "Synthesis under-filled, adding overflow to Autre"
        );

        // Count domain occurrences across all categories for source diversity enforcement
        let mut domain_counts: HashMap<String, usize> = HashMap::new();
        for items in all_scraped.values() {
            for item in items {
                if let Some(domain) = extract_domain(&item.url) {
                    *domain_counts.entry(domain).or_insert(0) += 1;
                }
            }
        }

        let max_per_source = settings.max_articles_per_source as usize;
        let mut added = 0usize;

        for article in all_overflow {
            if added >= shortfall {
                break;
            }
            // Enforce source diversity on overflow articles
            if let Some(domain) = extract_domain(&article.url) {
                let count = domain_counts.get(&domain).copied().unwrap_or(0);
                if count >= max_per_source {
                    continue;
                }
                *domain_counts.entry(domain).or_insert(0) += 1;
            }
            all_scraped
                .entry("category_autre".to_string())
                .or_default()
                .push(article);
            added += 1;
        }

        if added > 0 {
            tracing::info!(added = added, "Added overflow articles to Autre");
        }
    }

• Step 3: Add unit tests for fill-up calculation

    // ── fill-up calculation tests ───────────────────────────────

    #[test]
    fn fillup_target_calculation() {
        // 4 categories x 4 items = 16 max, 75% = 12
        let max = 4 * 4;
        let target = (0.75_f64 * max as f64).ceil() as usize;
        assert_eq!(target, 12);
    }

    #[test]
    fn fillup_shortfall_saturating() {
        // If total exceeds target, shortfall should be 0 (not panic)
        let target: usize = 12;
        let total: usize = 15;
        let shortfall = target.saturating_sub(total);
        assert_eq!(shortfall, 0);
    }

    #[test]
    fn classification_overflow_collected_when_all_full() {
        use crate::models::synthesis::ScrapedNewsItem;
        let articles: Vec<ScrapedNewsItem> = (0..6).map(|i| ScrapedNewsItem {
            title: format!("Art{}", i), url: format!("https://a.com/{}", i),
            summary: "s".into(), original_title: "t".into(), scraped_content: "c".into(),
        }).collect();
        let categories = vec!["AI News".to_string(), "Autre".to_string()];
        let response = serde_json::json!({
            "assignments": (0..6).map(|i| serde_json::json!({"index": i, "category": "AI News"})).collect::<Vec<_>>()
        });
        let mut filled = HashMap::new();
        let (result, overflow) = parse_classification_response(&response, &articles, &categories, 2, &mut filled);

        // AI News capped at 2, Autre gets 2 overflow, remaining 2 go to overflow vec
        assert_eq!(result.get("category_0").map(|v| v.len()), Some(2));
        assert_eq!(result.get("category_autre").map(|v| v.len()), Some(2));
        assert_eq!(overflow.len(), 2, "2 articles should overflow when both AI News and Autre are full");
    }

• Step 4: Run tests

Run: cd backend && cargo test --lib Expected: all tests pass

• Step 5: Commit

git add backend/src/services/synthesis.rs
git commit -m "feat: Autre fill-up to 75% synthesis target with source diversity enforcement"