You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
199 lines
8.3 KiB
Markdown
199 lines
8.3 KiB
Markdown
# AI Weekly Synth -- Architecture Document
|
|
|
|
## 1. System Overview
|
|
|
|
AI Weekly Synth is a self-hosted Rust/Axum backend with a SolidJS frontend, backed by PostgreSQL, deployed as a Docker Compose stack. It generates AI-powered weekly news syntheses organized by user-configured themes and categories.
|
|
|
|
See `requirements.md` for product vision and features. See `technical_specs.md` for the full technology stack. See `deployment.md` for the Docker topology and operational details.
|
|
|
|
---
|
|
|
|
## 2. Layer Architecture
|
|
|
|
The backend follows a three-layer architecture with shared model types:
|
|
|
|
```
|
|
handlers/ (HTTP layer)
|
|
│
|
|
services/ (Business logic)
|
|
│
|
|
db/ (Data access)
|
|
│
|
|
models/ (Shared types -- used by all layers)
|
|
```
|
|
|
|
Handlers extract and validate request data, delegate to services or db, and format responses. Services contain all business logic. The db layer executes pure SQL via sqlx with typed result mapping and no business logic. Models define domain structs, request/response DTOs, and validation logic.
|
|
|
|
See `dev_guidelines.md` Section 2 for complete project structure.
|
|
|
|
---
|
|
|
|
## 3. Key Components
|
|
|
|
### 3.1 LLM Provider Abstraction
|
|
|
|
The `LlmProvider` trait defines a unified interface for all LLM backends, with implementations for Gemini, OpenAI, Anthropic, and a mock provider for testing. A factory creates provider instances by name from the admin-curated provider list.
|
|
|
|
See `technical_specs.md` Section 6 for provider interface details and supported models.
|
|
|
|
### 3.2 Synthesis Pipeline
|
|
|
|
The pipeline is orchestrated in `services/synthesis.rs` and runs as a background tokio task with a 15-minute timeout. Phase 1 processes the user's personalized sources using a rolling windowed extraction with batched parallel scraping and LLM classification. Phase 2 fills remaining category gaps via Brave Search or LLM web search. The finalization step assembles sections, persists the synthesis, and records article history. Progress is reported via `tokio::sync::watch` channels consumed by SSE endpoints.
|
|
|
|
See `technical_specs.md` Section 5 for the full algorithm.
|
|
|
|
### 3.3 Job Store
|
|
|
|
`JobStore` (`services/job_store.rs`) is an in-memory concurrent store for active generation jobs:
|
|
|
|
- Backed by `DashMap<Uuid, JobEntry>` for lock-free access
|
|
- `DashSet<Uuid>` for per-user deduplication (one active job per user)
|
|
- Each job holds a `watch::Sender<ProgressEvent>` for real-time SSE streaming
|
|
- `AtomicBool` for cooperative cancellation
|
|
- 1-hour TTL with automatic cleanup
|
|
|
|
### 3.4 Scheduler
|
|
|
|
`services/scheduler.rs` runs as a background task checking every minute for due `theme_schedules`. When a schedule fires it runs the generation pipeline directly, emails results to configured recipients (up to 3), and marks the schedule as run to prevent double-execution on the same day.
|
|
|
|
See `deployment.md` for operational details.
|
|
|
|
### 3.5 Scraper
|
|
|
|
Two scraping services:
|
|
|
|
- **`scraper.rs`**: Article page scraper with SSRF prevention, HTML parsing, title/date/body extraction, soft-404 detection, 15s timeout, 5MB body limit.
|
|
- **`source_scraper.rs`**: Source index page scraper that extracts article links from user-configured source URLs (HTML `<a>` parsing with filters).
|
|
|
|
### 3.6 Rate Limiters
|
|
|
|
- **Auth rate limiter**: 10 requests/60s per key (email or IP) for magic link endpoints.
|
|
- **Provider rate limiter**: Per-LLM-provider sliding window, admin-configured, hot-reloaded from DB.
|
|
- **User rate limiters**: Per-user generation rate limits cached in `DashMap`, recreated on settings change.
|
|
|
|
---
|
|
|
|
## 4. Data Model
|
|
|
|
### Tables and Relationships
|
|
|
|
```
|
|
users
|
|
├── sessions (user_id FK, CASCADE)
|
|
├── magic_tokens (email reference, no FK)
|
|
├── settings (user_id PK/FK, CASCADE)
|
|
├── themes (user_id FK, CASCADE)
|
|
│ ├── sources (theme_id FK, CASCADE)
|
|
│ ├── syntheses (theme_id FK, SET NULL)
|
|
│ └── theme_schedules (theme_id FK, CASCADE, UNIQUE)
|
|
├── user_api_keys (user_id FK, CASCADE; UNIQUE per provider)
|
|
├── article_history (user_id FK, CASCADE)
|
|
├── llm_call_log (user_id FK, CASCADE)
|
|
└── audit_log (admin_user_id FK, SET NULL)
|
|
|
|
admin_providers
|
|
└── admin_rate_limits (provider_name FK, CASCADE)
|
|
```
|
|
|
|
See `technical_specs.md` Section 3 for complete column definitions.
|
|
|
|
---
|
|
|
|
## 5. API Overview
|
|
|
|
See `technical_specs.md` Section 4 for complete API endpoint specifications.
|
|
|
|
---
|
|
|
|
## 6. Security Architecture
|
|
|
|
### Authentication & Session Management
|
|
|
|
- **Passwordless**: Magic link tokens sent via email (Resend API), single-use, time-limited
|
|
- **Captcha**: Cloudflare Turnstile on registration and login
|
|
- **Sessions**: SHA-256 hashed tokens stored in DB, 30-day expiry, `HttpOnly` + `SameSite=Lax` cookies, optionally `Secure`
|
|
- **Anti-enumeration**: Same response for existent/non-existent emails, timing attack mitigation
|
|
- **Authorization**: `AuthUser` and `AdminUser` Axum extractors enforce auth levels per handler
|
|
|
|
### CSRF Protection
|
|
|
|
All mutating API endpoints require the `X-Requested-With` header (checked by `csrf::csrf_check` middleware layer). Non-mutating GET/HEAD/OPTIONS requests are exempt.
|
|
|
|
### Encryption at Rest
|
|
|
|
User LLM API keys are encrypted with AES-256-GCM before storage:
|
|
- 32-byte master key from `MASTER_ENCRYPTION_KEY` env var (64 hex chars)
|
|
- Random 12-byte nonce per encryption (stored alongside ciphertext)
|
|
- Key bytes are zeroized on drop (`zeroize` crate)
|
|
- Only a key prefix (first 8 chars + "...") is ever returned via the API
|
|
|
|
### SSRF Prevention
|
|
|
|
Both `scraper.rs` and `source_scraper.rs` validate URLs before fetching:
|
|
- DNS resolution check against private/loopback IP ranges
|
|
- Redirect chain validation (no redirects to private IPs)
|
|
- Only HTTP/HTTPS schemes allowed
|
|
|
|
### Security Headers
|
|
|
|
Applied as global middleware layers:
|
|
- `Content-Security-Policy` (self + Cloudflare Turnstile)
|
|
- `X-Content-Type-Options: nosniff`
|
|
- `X-Frame-Options: DENY`
|
|
- `Referrer-Policy: strict-origin-when-cross-origin`
|
|
- `X-XSS-Protection: 1; mode=block`
|
|
- `Strict-Transport-Security` (HTTPS only)
|
|
|
|
### Error Sanitization
|
|
|
|
The `sanitize_error_message` function strips API keys and internal details from error messages before they reach SSE clients. Internal errors log full details server-side but return generic messages to users.
|
|
|
|
### CORS
|
|
|
|
Configured to allow only the `APP_URL` origin, with credentials (cookies), limited to GET/POST/PUT/DELETE methods.
|
|
|
|
---
|
|
|
|
## 7. Concurrency Model
|
|
|
|
### Async Runtime
|
|
|
|
Tokio with full features. The Axum server runs as a multi-threaded async runtime.
|
|
|
|
### Background Tasks
|
|
|
|
Three tasks are spawned at startup: hourly session cleanup, periodic job store TTL cleanup, and the minute-by-minute theme schedule checker. See `deployment.md` Section 2.
|
|
|
|
### Generation Pipeline Concurrency
|
|
|
|
- **`tokio::task::JoinSet`**: Used for parallel scraping (bounded concurrency of 5 for source extraction) and parallel LLM classification calls within each batch
|
|
- **`tokio::sync::watch`**: Fan-out progress notifications to SSE clients; late subscribers immediately receive the latest state
|
|
- **`AtomicBool`**: Cooperative cancellation flag checked between pipeline stages; avoids mutex overhead
|
|
- **`DashMap` / `DashSet`**: Lock-free concurrent access for the job store (job entries), generating-users set, per-user rate limiter cache, and provider rate limiter state
|
|
|
|
### Task Lifecycle
|
|
|
|
```
|
|
POST /generate
|
|
└── handler creates job in JobStore
|
|
└── spawns outer task (panic monitor)
|
|
└── spawns inner task (15-min timeout)
|
|
└── run_generation_inner()
|
|
├── Phase 1 (JoinSet scrape, JoinSet classify)
|
|
├── Phase 2 (JoinSet scrape, JoinSet classify)
|
|
└── Save to DB
|
|
└── on complete/error: send final ProgressEvent
|
|
└── delayed cleanup (5 min) then remove from JobStore
|
|
```
|
|
|
|
### Graceful Shutdown
|
|
|
|
The server supports graceful shutdown via signal handling, allowing in-flight requests to complete.
|
|
|
|
---
|
|
|
|
## 8. Quality Gates
|
|
|
|
- Release candidates must include deterministic CI coverage for critical autonomous flows, especially scheduler execution and SSE progress behavior.
|
|
- External-provider tests (for example live LLM E2E checks) are supplemental and non-blocking; they do not replace deterministic CI coverage.
|