ai_synth/docs/team-analysis/04-devils-advocate.md

Devil's Advocate Analysis: AI Weekly Synth Refactoring

Preamble

This document exists to stress-test every assumption behind the proposed refactoring. The current application -- a React/Firebase/Gemini SPA -- is functional, deployed, and serving users. The proposal is to rewrite everything: new backend language (Rust), new database (SQLite), new frontend framework (SolidJS), new auth system, new LLM abstraction layer, and new deployment model. That is not a refactoring. That is a greenfield rewrite wearing a refactoring's name, and it deserves brutal scrutiny.

---

1. Questioning the Technology Choices

1.1 Rust for the Backend

The case against:

The current application has zero backend code. It is a thin SPA that talks to Firebase and Gemini directly from the browser. The "backend" being introduced serves three purposes: (a) proxy LLM API calls, (b) manage auth sessions, and (c) CRUD operations on a database. None of these require Rust.

• Development velocity: Rust's compile times, borrow checker friction, and verbose error handling will slow feature iteration by 3-5x compared to Go, Python/FastAPI, or Node.js/Express. This is a CRUD app with LLM API calls, not a systems programming project.
• Ecosystem maturity for web: The Rust web ecosystem (actix-web, axum, tower) is maturing but remains significantly less documented and less battle-tested for standard web applications than Express, FastAPI, or Go's net/http. Libraries for session management, email sending, CSRF protection, and OAuth flows are less polished.
• Hiring and maintenance: If the original developer leaves, finding a Rust web developer is substantially harder and more expensive than finding someone who knows Python, Go, or Node.js. The codebase becomes a bus-factor-1 liability.
• What Rust gives you that you do not need: Memory safety without a GC, zero-cost abstractions, and bare-metal performance. This application proxies LLM API calls that take 10-60 seconds each. The bottleneck is external API latency, not backend compute. You could run this on Python with FastAPI and it would perform identically from the user's perspective.

The question to ask: Is Rust chosen because it is the right tool, or because someone wants to learn Rust?

Alternative: Go gives you compiled binary deployment, excellent concurrency, trivial cross-compilation, a mature web ecosystem (Gin, Echo, Chi), and a learning curve measured in weeks rather than months. Python/FastAPI gives you the fastest development speed, native async, and the richest LLM SDK ecosystem (every LLM provider ships a Python SDK first).

1.2 SolidJS for the Frontend

The case against:

The current React frontend is approximately 900 lines of code across 7 files. It is straightforward, uses standard patterns (Context, useEffect, useState), and works. The proposal is to rewrite all of it in SolidJS.

• Cost/benefit ratio: SolidJS's reactive model is technically superior to React's virtual DOM diffing. But for an app with 5 pages, no complex animations, no heavy lists, and no real-time data visualization, the performance difference is imperceptible. You are paying the full cost of a rewrite for zero user-visible benefit.
• Ecosystem: SolidJS has roughly 1/100th of React's ecosystem. lucide-react does not have a SolidJS version (you would need lucide-solid or raw SVGs). react-router-dom becomes @solidjs/router with different APIs. date-fns works, but any React-specific utility does not.
• Tailwind CSS: Works with SolidJS, but the JSX-like syntax (JSX vs TSX) is slightly different (e.g., class vs className, for vs htmlFor). Every component needs manual translation.
• Real-time listeners: The current app uses Firestore's onSnapshot for real-time updates on the Home and Sources pages. SolidJS's reactive primitives (createSignal, createEffect) differ fundamentally from React's useState/useEffect. The subscription cleanup patterns are different. This is not a find-and-replace operation.
• Developer pool: If you hire someone to maintain this, they probably know React. They probably do not know SolidJS.

The question to ask: What is SolidJS giving you that React 19 with its compiler cannot? Name one user-facing feature that is impossible or unreasonably difficult in React but easy in SolidJS.

Alternative: Keep React. The frontend works. If you want to modernize, adopt React Server Components or Next.js/Remix. If you want a different framework for legitimate reasons, consider Svelte (larger ecosystem than SolidJS, similar reactivity model, larger community).

1.3 SQLite with "Postgres Upgrade Path"

The case against:

• Write concurrency: SQLite uses a single-writer model. When user A is generating a synthesis (which involves multiple writes during scraping/validation), user B's writes will block. With WAL mode, reads are concurrent, but writes serialize. For a "multi-user" application, this is a real limitation that manifests at surprisingly low user counts (10-20 concurrent writers).
• The "upgrade path" is a myth: sqlx supports both SQLite and Postgres, but the SQL dialects differ. SQLite lacks: JSONB operators, ARRAY types, LATERAL JOIN, INSERT ... ON CONFLICT DO UPDATE (supported since 3.24 but with different syntax), ENUM types, advisory locks, and LISTEN/NOTIFY. If you write queries that use any Postgres-specific feature, you cannot "downgrade" to SQLite. If you restrict yourself to the SQLite subset, you are not getting the benefits of Postgres. You end up writing to the lowest common denominator of both databases.
• Migration tooling: sqlx migrations work for both, but you will need separate migration files if the dialects diverge. Testing requires running against both databases. CI doubles.
• Operational burden: SQLite is a file on disk. Backups are cp. But that means backup-during-write requires VACUUM INTO or filesystem snapshots. There is no pg_dump-equivalent streaming backup. There is no replication. There is no point-in-time recovery.

The question to ask: How many concurrent users are expected? If more than 10-20, start with Postgres. If fewer than 5, SQLite is fine, but then why plan for a "Postgres upgrade path" you will never need?

Alternative: Just use Postgres from the start. Docker makes this trivial. A docker-compose.yml with Postgres + the app is no more complex than SQLite, and you avoid the "upgrade path" fiction entirely.

---

2. Challenging the Scope

2.1 This Is a Complete Rewrite, Not a Refactoring

Let me enumerate what is being changed:

Component	Current	Proposed	Effort
Backend	None (serverless)	Rust + axum/actix	Build from scratch
Database	Firestore (managed)	SQLite/Postgres (self-managed)	Build from scratch
Frontend framework	React 19	SolidJS	Complete rewrite
Auth	Firebase Auth (Google SSO)	Email+captcha, magic link, sessions	Build from scratch
LLM integration	Gemini only (frontend)	Multi-provider (backend)	Major redesign
Deployment	Firebase Hosting	Docker/VM	Build from scratch
Admin module	None	New feature	Build from scratch
Email system	Gmail API (frontend)	Backend email service	Build from scratch

Every single component is being replaced. There is no code that survives this refactoring. Zero reuse. This is the textbook definition of a "Second System Effect" (Brooks, 1975).

2.2 What Is Actually Broken?

Before rewriting everything, let us be honest about what problems exist today:

1. API key exposure: The Gemini API key is bundled in the frontend JavaScript. This is a real security issue, but it can be fixed with a single Cloud Function or a tiny proxy server -- no full rewrite needed.
2. CORS proxy reliance: The scraping uses third-party CORS proxies that can go down. A backend proxy fixes this, but again, a single endpoint on any lightweight server suffices.
3. Vendor lock-in: Firebase Authentication + Firestore. This is a legitimate concern for a self-hosted deployment, but the migration path is incremental: swap Firestore for Supabase (Postgres + real-time), keep Firebase Auth temporarily.
4. Single LLM provider: Only Gemini is supported. Adding providers is valuable, but can be done incrementally behind the existing frontend.

None of these problems require rewriting the frontend. None require Rust. None require SolidJS.

2.3 What Happens If the Project Is Abandoned Halfway?

With the current approach (total rewrite), abandonment at 50% means:

• The old app still runs on Firebase, but receives no updates.
• The new app has a partially built Rust backend, a half-ported SolidJS frontend, and no working deployment.
• Total investment: wasted.

With an incremental approach, each phase delivers value independently:

• Phase 1 (add backend proxy): Old app gets secure API key handling + server-side scraping. Value delivered even if Phase 2 never happens.
• Phase 2 (multi-LLM): Backend supports multiple providers. Value delivered even if Phase 3 never happens.
• Phase 3 (replace Firebase): Database migration to self-hosted. Value delivered even if Phase 4 never happens.
• Phase 4 (frontend modernization): If still needed.

2.4 Realistic Timeline Estimate

For a single developer working part-time on this:

Task	Optimistic	Realistic	Pessimistic
Rust backend skeleton (auth, CRUD, LLM proxy)	3 weeks	6 weeks	12 weeks
Multi-provider LLM abstraction	2 weeks	4 weeks	8 weeks
Auth system (email, magic link, sessions, captcha)	2 weeks	5 weeks	10 weeks
SolidJS frontend rewrite (5 pages + components)	2 weeks	4 weeks	8 weeks
Admin module	1 week	2 weeks	4 weeks
Data migration tooling	1 week	2 weeks	3 weeks
Docker/deployment	1 week	2 weeks	3 weeks
Testing and debugging	2 weeks	4 weeks	8 weeks
Total	14 weeks	29 weeks	56 weeks

That is 7 months realistically, over a year pessimistically. For an application that is currently approximately 1,200 lines of TypeScript.

---

3. Hidden Complexity

3.1 Multi-Provider LLM Abstraction: The Hardest Problem Nobody Is Talking About

The current pipeline in geminiService.ts is deeply coupled to Gemini's specific capabilities:

1. Pass 1 uses googleSearch grounding -- Gemini searches the web in real-time and returns grounded results with citations. This is not a generic "LLM call." It is a tool-use pattern specific to Gemini's architecture.
2. Structured output uses responseSchema with Type enums from @google/genai. This forces JSON output conforming to a schema. OpenAI has response_format: { type: "json_schema" }. Anthropic has tool-use-based structured output. They are similar but not identical.
3. Pass 2 is a standard completion call. This is the easy part to abstract.

The fundamental problem: Gemini's googleSearch grounding is not just "web search." It is integrated into the model's reasoning process. The model sees search results inline and can refine its queries. To replicate this with OpenAI, you would need:

• OpenAI with browsing (ChatGPT-only, not available via API for all models).
• Or: A separate web search API (Serper, Brave, Google Custom Search) + a tool-use loop where the model calls search, reads results, decides to search again, etc.

With Anthropic, you would need:

• Claude with the web_search tool (available but with different result formatting).
• Or: Same external search + tool-use loop.

What gets lost in abstraction: Each provider returns search results differently. Gemini's grounded responses include inline citations with confidence scores. OpenAI's browsing returns rendered page content. Anthropic's web search returns snippets. The quality and structure of Pass 1 output will vary dramatically by provider, and Pass 2's rewriting quality depends on Pass 1's quality.

The question to ask: Are you building an abstraction over fundamentally different capabilities, or are you building three separate pipelines that happen to share a common interface? The latter is honest but expensive. The former is a leaky abstraction waiting to break.

3.2 The Scraping Pipeline

The current scraping in filterValidNewsItems runs in the browser using DOMParser. It:

• Fetches HTML via CORS proxies.
• Parses the DOM for meta tags, JSON-LD, <time> elements.
• Detects soft 404s by scanning title/h1 content.
• Extracts body text (up to 4,000 chars) after removing scripts/nav/footer.

Moving this server-side is straightforward (use reqwest + scraper or select.rs in Rust), but consider:

• Rate limiting against target sites: Server-side scraping from a single IP is more likely to be blocked than distributed browser requests via CORS proxies.
• JavaScript-rendered content: Many news sites render content via JavaScript. The current approach works because CORS proxies often return server-rendered HTML. A Rust backend would need a headless browser (Chromium via headless_chrome or fantoccini) for JS-heavy sites, adding significant complexity and resource usage.
• Legal considerations: Server-side scraping at scale may trigger ToS violations. The current approach is effectively each user's browser making requests through proxies, which is more defensible.

3.3 Authentication: You Are Building a Small Identity Provider

The proposed auth system includes:

• Email + captcha account creation
• Magic link / passwordless auth
• Session-based auth with secure cookies

This is not a feature. This is building a mini identity provider. Let me enumerate the hidden work:

1. Email delivery infrastructure: Magic links require sending emails. Self-hosted SMTP is a minefield of deliverability issues (SPF, DKIM, DMARC, IP reputation, bounce handling). You will use a transactional email service (SendGrid, Postmark, SES). That is a new dependency, a new account, a new API integration, and a new failure mode.
2. Token management: Magic link tokens must be cryptographically secure, single-use, time-limited, and stored server-side. This is a custom token store with cleanup jobs.
3. Session management: Secure cookies require HTTPS (certificate management), proper SameSite/Secure/HttpOnly flags, session storage (in-memory? database? Redis?), session rotation on privilege changes, and idle/absolute timeout handling.
4. Captcha integration: reCAPTCHA or hCaptcha requires a frontend widget + server-side verification. Another API key to manage.
5. Account recovery: What happens when a user loses access to their email? Password reset? Account deletion? GDPR compliance?
6. Rate limiting on auth endpoints: Brute-force protection, account enumeration prevention, IP-based throttling.
7. Email verification: Is the email valid? Bounce handling? Typo detection?

Comparison: Firebase Auth gives you all of this for free, plus Google SSO, Apple SSO, email/password auth, phone auth, anonymous auth, multi-factor auth, and is SOC2/ISO27001 compliant. You are proposing to replace a battle-tested managed service with a hand-rolled implementation.

Alternative: Use a self-hosted auth service like Authentik, Keycloak, or Authelia. These are deployable via Docker, support email+password, magic link, OAuth, and session management out of the box. Zero custom auth code needed.

3.4 Admin Module

The admin module for "LLM API key configuration" and "rate limiter configuration" sounds simple, but implies:

• Role-based access control (who is an admin?)
• A secure key storage mechanism (encrypting API keys at rest)
• A UI for managing per-provider rate limits
• Runtime configuration changes (changing rate limits without restart)
• Audit logging (who changed what, when?)

This is not a settings page. This is an operational control plane.

---

4. Spotting the Contradictions

4.1 "Single VM deployment" + "with or without Docker"

• With Docker: Straightforward. docker build produces a container, docker-compose adds Postgres. Works anywhere Docker runs.
• Without Docker: You need to compile Rust for the target architecture. If developing on macOS (darwin/arm64) and deploying to Linux (amd64), you need cross-compilation via cross or a CI/CD pipeline that builds on Linux. Rust cross-compilation is possible but non-trivial (especially with OpenSSL/TLS dependencies). You also need to install and configure SQLite/Postgres natively, manage the SolidJS build output, set up a reverse proxy (nginx/caddy), and handle process management (systemd).

The question to ask: Is "without Docker" a hard requirement or a nice-to-have? If the latter, drop it and save significant deployment complexity.

4.2 "Preserve UX and look & feel" + "switch to SolidJS"

The current app uses:

• React Context for auth state propagation.
• useEffect + onSnapshot for real-time Firestore listeners with cleanup.
• react-router-dom v7 for routing with <Link>, useParams, useNavigate.
• Controlled form inputs with useState.
• Conditional rendering patterns ({condition && <Component />}).

SolidJS equivalents exist, but they work differently:

• createContext exists but components do not re-render -- signals propagate granularly. This changes how you structure state.
• createEffect does not take a dependency array. It auto-tracks. Cleanup uses onCleanup. The mental model is different.
• @solidjs/router has <A> instead of <Link>, useParams() returns a reactive proxy, not a plain object.
• Signals replace useState. createSignal returns [getter, setter] where the getter is a function, not a value. Every template reference becomes count() instead of count.
• Conditional rendering uses <Show> and <For> components, not inline JSX ternaries (which would defeat fine-grained reactivity).

"Preserving UX" means the user sees the same screens. "Preserving look and feel" means Tailwind classes copy over. But the underlying component logic must be substantially rewritten. This is not a port; it is a rewrite guided by screenshots.

4.3 "Improve CORS proxy handling"

The current app uses CORS proxies because it scrapes from the browser. A backend can scrape directly. So the "improvement" is not improving CORS handling -- it is eliminating the need for CORS proxies entirely. This is a side effect of adding a backend, not a feature to list separately. Listing it as a requirement inflates the apparent scope and obscures the real motivation.

4.4 "Remove Google hosting dependencies" + Email-based Auth Still Needs an External Service

You are removing dependency on Firebase (Google) but adding dependency on:

• A transactional email provider (SendGrid/Postmark/SES) for magic links.
• A captcha provider (Google reCAPTCHA or hCaptcha).
• Potentially a web search API (Serper/Brave/Google CSE) for non-Gemini providers.

You are trading one set of dependencies for another. The net dependency count may actually increase.

---

5. Risk Assessment

Top 5 Risks That Could Derail This Project

1. Scope creep leading to abandonment: The project is already at maximum scope (full rewrite + new features). Any scope creep (even small -- "oh, we also need password auth" or "can we add dark mode?") pushes the timeline past the point where motivation sustains the effort. For a part-time/side project, this is the #1 killer.

2. Multi-LLM abstraction turns into a quagmire: The web search grounding capabilities differ so fundamentally across providers that the abstraction layer either (a) drops to lowest common denominator (losing Gemini's best feature), (b) becomes provider-specific spaghetti code behind a thin interface, or (c) consumes weeks of engineering time trying to build equivalence that does not exist.

3. Auth system vulnerabilities: Hand-rolled authentication is the most common source of security vulnerabilities in web applications. Missing CSRF protection, insecure token storage, session fixation, timing attacks on token comparison, missing rate limiting -- any of these creates a vulnerability that Firebase Auth handled for free.

4. Data migration failures: Existing users have data in Firestore (syntheses, sources, settings). Migrating this to SQLite/Postgres requires: (a) exporting from Firestore, (b) transforming the schema (NoSQL to relational), (c) handling the legacy SynthesisData format (with both old field-based and new sections[]-based structures), and (d) validating that nothing was lost. This is tedious, error-prone work that is easy to underestimate.

5. Deployment complexity kills adoption: The current app deploys to Firebase Hosting with firebase deploy. The new app requires: (a) compiling Rust, (b) building SolidJS, (c) configuring a reverse proxy, (d) managing a database, (e) setting up email delivery, (f) managing TLS certificates, and (g) process supervision. Even with Docker, this is an order of magnitude more operational burden. For a personal/small-team tool, operational burden directly reduces the likelihood of the app surviving.

Rollback Plan

There is no rollback plan because there is nothing to roll back to. The old app continues to run on Firebase until it is deliberately shut down. The risk is not "rolling back" but rather "having spent months on a new version that never reaches feature parity with the original."

Impact on Existing Users and Data

• Existing users must create new accounts (no Firebase Auth = no Google SSO migration path unless you also implement Google OAuth in the new system).
• Data must be explicitly migrated from Firestore to the new database.
• The email feature currently uses Gmail API via OAuth popup. The new system would need its own email sending capability.
• If the migration is botched, user data is lost. Firestore has no "undo."

---

6. Alternative Approaches

6.1 Minimal Viable Refactoring (2-4 weeks)

Keep React. Keep Firestore. Add a tiny backend (Node.js/Express or Python/FastAPI) that:

1. Proxies Gemini API calls (solves the API key exposure problem).
2. Handles scraping server-side (eliminates CORS proxy dependency).
3. Exposes an endpoint for email sending (replaces Gmail API popup).

What this gives you: The two biggest security/reliability issues solved, with zero frontend changes, zero database migration, and zero user impact. Deploy the proxy on a small VM or as a Cloud Function.

6.2 Incremental Backend Migration (3-6 months, phased)

Phase 1 -- API Gateway (4 weeks):

• Build a lightweight backend (Go or Python) that proxies LLM calls and handles scraping.
• Frontend calls the new backend instead of Gemini directly.
• Firebase Auth + Firestore remain.
• Deliverable: Secure API key handling, reliable scraping.

Phase 2 -- Multi-Provider LLM (4 weeks):

• Backend supports Gemini, OpenAI, Anthropic.
• Settings page adds provider selection.
• Provider-specific adapters behind a common interface.
• Deliverable: LLM provider choice.

Phase 3 -- Database Migration (4 weeks):

• Replace Firestore with Postgres (via Supabase for managed, or raw Postgres for self-hosted).
• Write migration scripts.
• Update frontend Firestore calls to REST API calls.
• Deliverable: No more Firebase dependency for data.

Phase 4 -- Auth Migration (4 weeks):

• Replace Firebase Auth with self-hosted auth (Authentik/Keycloak) or add email auth alongside Google OAuth.
• Deliverable: Self-hosted auth.

Phase 5 -- Frontend Modernization (optional, 4 weeks):

• If React is genuinely insufficient, consider Svelte or keep React.
• Deliverable: Modernized frontend (if needed).

Each phase ships independently. Each phase delivers value. Abandonment at any phase leaves the system in a better state than it started.

6.3 For Each Major Decision: Alternatives and Trade-offs

Decision	Proposed	Alternative 1	Alternative 2
Backend language	Rust	Go: faster dev, easy cross-compile, good web ecosystem	Python/FastAPI: fastest dev, best LLM SDK ecosystem, async
Frontend framework	SolidJS	Keep React: zero rewrite cost, huge ecosystem	Svelte: similar reactivity to SolidJS, larger community
Database	SQLite + "Postgres path"	Postgres from start: no migration debt	Supabase: managed Postgres + real-time + auth
Auth	Custom email+magic link	Authentik/Keycloak: self-hosted, full-featured, zero custom code	Keep Firebase Auth: works, managed, free tier
Email delivery	Custom implementation	Postmark/SendGrid: managed, deliverable, $0-20/month	Keep Gmail API: already works, zero infrastructure
Deployment	Docker on VM	Fly.io/Railway: managed Docker, free tier, simpler ops	Keep Firebase Hosting + add Cloud Functions

---

7. Hard Questions for the Project Owner

These are questions where the answer significantly changes the implementation. They must be answered before writing a single line of code.

1. How many users will this serve? If < 5 (personal/team tool), SQLite is fine, self-hosted auth is overkill, and operational simplicity should dominate every decision. If > 50, you need Postgres, proper auth, and monitoring from day one.

2. Is the Rust choice driven by the project's needs or by a learning goal? Both answers are valid, but they lead to different architectures. A learning project should embrace complexity. A product should minimize it.

3. What is the budget for external services? Transactional email ($0-20/month), LLM APIs ($10-100/month), domain + TLS ($10/year), VM ($5-50/month). If budget is zero, Firebase's free tier is hard to beat.

4. What happens to existing Firestore data? Is there a migration plan? How many syntheses exist? Are the legacy format syntheses (with majorAnnouncements, financialSector, etc.) being migrated or abandoned?

5. Is Google SSO being kept alongside the new auth methods, or replaced entirely? If replaced, existing users lose access to their accounts. If kept, you need OAuth2 client implementation in the new backend.

6. Which LLM providers are actually needed on day one? If the answer is "just Gemini for now, but we want the option for others," build the abstraction later. YAGNI applies. Adding a second provider later is cheaper than building a three-provider abstraction upfront that may never be used.

7. Who will operate this in production? Self-hosted means self-operated. Database backups, TLS certificate renewal, security patches, log monitoring, uptime monitoring. Is someone committed to this ongoing work?

8. Is "without Docker" a hard requirement? Supporting both Docker and non-Docker deployment doubles the deployment documentation and testing surface. If Docker is available on the target VM, mandate it and simplify.

9. What is the desired SLA? If the app can be down for hours without consequence (personal tool), the architecture can be simpler. If uptime matters, you need health checks, restart policies, and monitoring.

10. How will the admin module's API keys be secured at rest? SQLite stores data in a plain file. API keys in SQLite are readable by anyone with file access. Do you need encryption at rest? If so, how? (SQLCipher? Application-level encryption? OS-level disk encryption?)

11. What is the testing strategy? The current app has zero tests. The proposed rewrite has zero mention of testing. A Rust backend should have integration tests for every endpoint, unit tests for the LLM abstraction, and end-to-end tests for the auth flow. Is this budgeted?

12. Is the 2-pass generation pipeline (search + scrape + rewrite) non-negotiable? Some LLM providers may not support grounded web search at all. If the pipeline is mandatory, this constrains which providers are viable. If Pass 1 can be simplified (e.g., just use a search API + LLM summary without scraping), the multi-provider problem becomes much simpler.

13. What is the target response time for synthesis generation? Currently, generation takes 30-60+ seconds (two Gemini calls + parallel scraping). Will users accept the same latency? Should generation be asynchronous (start generation, get notified when done)?

14. Is the Gmail email sending feature being kept? The current implementation uses a Gmail OAuth popup to send via the user's own Gmail account. The new system would presumably send from a system email address. These are fundamentally different features with different privacy implications.

15. What is the concrete timeline and deadline? Without a deadline, scope expands indefinitely. With a deadline, scope must be ruthlessly cut. Which features are must-have for v1, and which are nice-to-have for v2?

---

8. Summary Verdict

The proposed refactoring is ambitious to the point of being reckless. It simultaneously changes every layer of the stack, introduces multiple new features, and removes all existing infrastructure -- with no phased delivery plan and no fallback position.

The current application has real problems (API key exposure, CORS proxy fragility, vendor lock-in), but none of them require a full rewrite. A 200-line proxy server solves the two most critical issues. An incremental migration plan addresses vendor lock-in without risking the entire project.

My recommendation: Before writing any code, answer the 15 questions above. Then build the smallest possible backend that solves the API key and scraping problems. Ship that. Use it for a month. Then decide if the rest of the rewrite is still justified, with the benefit of real experience operating a backend for this application.

The best refactoring is the one that delivers value at every intermediate step, not the one that promises everything at the end.