From 7ae6ef7ef1d12a3463ed88856d9f8ae6ca8cdd88 Mon Sep 17 00:00:00 2001 From: oabrivard Date: Sat, 21 Mar 2026 16:10:39 +0100 Subject: [PATCH] Defined implementation plan --- docs/implementation-plan/01-phase-roadmap.md | 608 ++++ docs/implementation-plan/02-backend-plan.md | 3189 +++++++++++++++++ docs/implementation-plan/03-frontend-plan.md | 2654 ++++++++++++++ .../04-infrastructure-plan.md | 2418 +++++++++++++ 4 files changed, 8869 insertions(+) create mode 100644 docs/implementation-plan/01-phase-roadmap.md create mode 100644 docs/implementation-plan/02-backend-plan.md create mode 100644 docs/implementation-plan/03-frontend-plan.md create mode 100644 docs/implementation-plan/04-infrastructure-plan.md diff --git a/docs/implementation-plan/01-phase-roadmap.md b/docs/implementation-plan/01-phase-roadmap.md new file mode 100644 index 0000000..59b575d --- /dev/null +++ b/docs/implementation-plan/01-phase-roadmap.md @@ -0,0 +1,608 @@ +# Phased Delivery Roadmap: AI Weekly Synth Rewrite + +**Date**: 2026-03-21 +**Author**: Phase Roadmap Planner +**Input**: Team analysis (01-04) and project decisions (05) + +--- + +## Overview + +This roadmap decomposes the AI Weekly Synth rewrite into 7 phases. Each phase produces a working, deployable application. The phases are ordered to deliver value incrementally: Phase 1 proves the stack works end-to-end, and each subsequent phase adds exactly one major capability. + +The decisions document establishes: Rust (Axum) + Postgres + SolidJS + Tailwind CSS, all 3 LLM providers, user-provided API keys, email + magic link auth, Docker-only deployment, no data migration, and testing as part of the plan. + +--- + +## Dependency Graph + +``` +Phase 1: Foundation (Axum + Postgres + SolidJS + Auth + Settings CRUD) + | + +---> Phase 2: Sources CRUD + Scraper Service + | | + | +---> Phase 4: LLM Provider Abstraction (Gemini first) + | | + | +---> Phase 5: Generation Pipeline + SSE Progress + | | | + | | +---> Phase 6: Multi-Provider (OpenAI + Anthropic) + | | + +---> Phase 3: Admin Module (Provider/Model Curation, Rate Limits) + | | + | +---> Phase 4 (admin curates provider list that Phase 4 uses) + | + +-------------------------------+---> Phase 7: Email (Resend) + Export (PDF/Markdown) +``` + +Summary of dependencies: +- Phase 2 depends on Phase 1 (auth, DB, frontend scaffolding) +- Phase 3 depends on Phase 1 (auth with admin role, DB) +- Phase 4 depends on Phase 2 (scraper) + Phase 3 (admin-curated provider/model list) +- Phase 5 depends on Phase 4 (LLM provider working) +- Phase 6 depends on Phase 5 (pipeline working with one provider) +- Phase 7 depends on Phase 1 only (can be started after Phase 1, but best done after Phase 5 so there is content to email/export) + +--- + +## Risk-Ordered Priority + +If time runs out, this is the order of criticality (most critical first): + +1. **Phase 1 -- Foundation**: Without this, nothing works. This is the riskiest phase because it involves setting up Rust/Axum from scratch (learning curve), hand-rolling auth (magic links, sessions, captcha), and standing up the SolidJS frontend with routing and auth context. Everything depends on this. + +2. **Phase 5 -- Generation Pipeline + SSE**: This is the core value proposition of the application. Without synthesis generation, the app is a settings manager. + +3. **Phase 4 -- LLM Provider Abstraction (Gemini)**: Prerequisite for Phase 5. Getting one provider working end-to-end with structured output and web search grounding proves the LLM integration works. + +4. **Phase 2 -- Sources CRUD + Scraper**: Sources feed the generation pipeline. The scraper is used during generation to validate URLs. + +5. **Phase 3 -- Admin Module**: Users cannot configure providers/models without this. However, for a single-user self-hosted scenario, environment variables or seed data could serve as a temporary workaround. + +6. **Phase 6 -- Multi-Provider**: Adds OpenAI and Anthropic. High value but the app is fully functional with Gemini alone. + +7. **Phase 7 -- Email + Export**: Nice-to-have features. The app works without them. Users can copy-paste or screenshot. + +--- + +## Phase 1: Foundation + +### Goal +Prove the entire stack works end-to-end: Rust serving a SolidJS SPA, Postgres connected, email + magic link authentication functioning, and one complete CRUD flow (user settings). + +### Deliverables + +**Backend (Rust/Axum)**: +- Project scaffold: Cargo workspace, `main.rs`, config loading (dotenvy), tracing/logging setup +- Postgres connection pool (sqlx) with compile-time checked queries +- Database migrations: `users`, `sessions`, `magic_link_tokens`, `settings` tables +- Unified error handling (`AppError` enum with `IntoResponse`) +- Auth system: + - `POST /api/v1/auth/register` -- email + Cloudflare Turnstile captcha validation, sends magic link via Resend + - `POST /api/v1/auth/login` -- request magic link (same response whether email exists or not) + - `GET /api/v1/auth/verify?token=...` -- verify token, create session, set cookie, redirect to app + - `POST /api/v1/auth/logout` -- invalidate session, clear cookie + - `GET /api/v1/auth/me` -- return current user info +- Session middleware: cookie extraction, SHA-256 lookup, expiration check (30-day), user injection into request extensions +- CSRF protection: `X-Requested-With` header check on mutating requests + `SameSite=Lax` cookies +- Rate limiting on auth endpoints (per-IP, per-email) +- Settings CRUD: `GET /api/v1/settings`, `PUT /api/v1/settings` +- CLI command: `create-admin` to bootstrap the first admin account +- Static file serving: serve the SolidJS build output from the Axum binary +- Security headers (CSP, X-Content-Type-Options, X-Frame-Options, HSTS, Referrer-Policy) +- CORS configuration + +**Frontend (SolidJS)**: +- Vite + SolidJS + TypeScript + Tailwind CSS project scaffold +- Auth context with signals (session check via `GET /api/v1/auth/me` on load) +- Route guard (redirect to `/login` if unauthenticated) +- Login page: email field, Turnstile widget, "Recevoir un lien de connexion" button, "Creer un compte" link +- Sign-up page: email + optional display name + Turnstile + "Creer mon compte" button +- Magic link confirmation screen ("Verifiez votre boite de reception", resend with cooldown) +- Navbar: logo, nav links (Syntheses, Sources), user email, Settings gear, Logout +- Mobile hamburger menu +- Active route indicator on nav links +- Settings page: theme, max age days, categories (dynamic list with add/remove), max items per category, search agent behavior, AI model dropdown (hardcoded placeholder for now) +- Error boundary (top-level `ErrorBoundary` component) +- Session expiry handling (401 -> redirect to login with message) +- i18n-ready structure: all user-facing strings in a central `fr.ts` locale file, accessed via a helper function + +**Infrastructure**: +- `Dockerfile` (multi-stage: Rust build + SolidJS build -> minimal runtime image) +- `docker-compose.yml` with Postgres service + app service +- `.env.example` with all required environment variables documented +- Resend integration for sending magic link emails + +**CLI**: +- `./ai-synth create-admin admin@example.com` creates an admin user (no magic link needed, account is pre-verified) + +### Definition of Done +- `docker compose up` starts the app with Postgres +- A new user can sign up (receives magic link email via Resend), click the link, and land on the home page +- The admin can be created via CLI +- Authenticated user can view and update their settings +- Unauthenticated requests return 401 +- Session persists across browser restarts (within 30-day window) +- Logout invalidates the session server-side +- Turnstile captcha prevents automated signups +- All pages render correctly on desktop and mobile + +### Dependencies +None -- this is the first phase. + +### Risk Factors +1. **Hand-rolled auth is the highest-risk component**. Magic link token lifecycle (generation, SHA-256 hashing, single-use enforcement, expiration, email enumeration prevention) must be implemented correctly from the start. A subtle bug here creates a security vulnerability. +2. **Rust learning curve**. Async Rust with Axum, sqlx, and tower middleware is non-trivial for someone learning Rust. Expect the borrow checker, lifetime annotations, and trait bounds to slow things down significantly in this phase. +3. **Email deliverability**. Magic links only work if emails arrive. Resend handles SPF/DKIM/DMARC, but initial setup, domain verification, and inbox placement testing can take time. +4. **Cloudflare Turnstile integration**. Requires server-side verification of the captcha token. The API is simple, but handling failures (network issues, invalid tokens, expired tokens) needs careful error UX. + +### Testing Scope +- **Unit tests**: Config parsing, session token generation/hashing, magic link token lifecycle, CSRF header validation, settings validation (serde + validator), `AppError` response formatting +- **Integration tests**: Full auth flow (register -> verify -> me -> logout), settings CRUD with auth, admin CLI command, rate limiting on auth endpoints (verify lockout after N failures), 401 on unauthenticated access +- **Frontend**: Manual smoke testing of all screens and flows (automated E2E testing deferred to a later phase when there is more to test) + +### Milestones +1. **M1.1 -- Rust skeleton compiles and serves "Hello World"**: Axum router, tracing, config loading, Postgres pool connected, migrations run. Docker build works. +2. **M1.2 -- Auth flow works end-to-end**: Register, magic link email sent (Resend), verify token, session cookie set, `GET /me` returns user, logout clears session. CLI create-admin works. +3. **M1.3 -- SolidJS shell renders**: Login page, navbar, settings page (static, no API calls yet). Tailwind styling matches the current app's visual language. Mobile hamburger menu works. +4. **M1.4 -- Frontend + backend integrated**: SolidJS auth context calls the API. Login/signup flow works through the UI. Settings page reads/writes via the API. Session expiry redirects to login. +5. **M1.5 -- Tests and hardening**: Unit and integration tests pass. Security headers configured. CSRF protection tested. Rate limiting on auth endpoints verified. + +--- + +## Phase 2: Sources CRUD + Scraper Service + +### Goal +Add the custom sources management feature (CRUD, bulk import, CSV import/export) and build the URL scraper service that will be used during synthesis generation. + +### Deliverables + +**Backend**: +- Database migration: `sources` table +- Sources API: + - `GET /api/v1/sources` -- list user's sources + - `POST /api/v1/sources` -- add a single source (title + URL, validated) + - `DELETE /api/v1/sources/:id` -- delete a source (ownership check) + - `POST /api/v1/sources/bulk` -- bulk import (JSON array) + - `POST /api/v1/sources/import-csv` -- CSV import (multipart upload) + - `GET /api/v1/sources/export-csv` -- CSV export download +- Input validation: URL format validation, title length limits, max sources per user +- Scraper service (`services/scraper.rs`): + - `reqwest` HTTP client (shared from `AppState`, with timeouts: 5s connect, 15s response, 30s total) + - SSRF prevention: DNS resolution check against private IP ranges, protocol restriction (http/https only), redirect validation + - HTML parsing with `scraper` crate: soft-404 detection, publication date extraction (meta tags, JSON-LD, `