knowfoolery/claude-plan-phase-1.md

# Progressive Phase 1 Implementation Plan: Incremental Infrastructure Setup

## Overview
Instead of setting up all infrastructure at once, we'll progressively build Phase 1 in three distinct sub-phases, allowing for thorough testing and validation at each layer before adding complexity.

## Phase 1A: Backend + Database Foundation (5-7 days)

### Core Infrastructure Setup
**Dependencies: None**

**1A.1 Go Project Structure & Basic Services**
- Create complete Go module structure for all microservices
- Implement basic Fiber apps with health endpoints for each service
- Set up shared packages structure (auth, database, observability, security)
- Create basic service discovery and configuration management
- Add graceful shutdown and signal handling

**1A.2 Database Layer with Ent**
- Define complete Ent schemas (Question, GameSession, QuestionAttempt)
- Set up dual database configuration (SQLite for dev, PostgreSQL for testing)
- Create migration scripts and database initialization
- Implement basic CRUD operations for each entity
- Add database connection pooling and health checks

**1A.3 Mock Authentication System**
- Create simple JWT-based mock authentication (no external calls)
- Implement basic role-based access control (player/admin)
- Add middleware for token validation using local secret
- Create user context propagation through services
- Add basic security headers and CORS

**1A.4 Basic Docker Environment**
- Create `docker-compose.basic.yml` with only PostgreSQL
- Add database initialization and seeding containers
- Create health check scripts for database connectivity
- Set up volume management for persistent data

**1A.5 Integration Testing Framework**
- Set up testcontainers for PostgreSQL integration tests
- Create test utilities for database seeding and cleanup
- Implement service-level integration tests
- Add API endpoint testing with mock authentication
- Create test data factories and fixtures

**Success Criteria for 1A:**
- All services start and respond to health checks
- Database migrations run successfully
- Mock authentication validates tokens correctly
- Integration tests pass with PostgreSQL container
- Services can perform basic CRUD operations
- Docker environment runs consistently

---

## Phase 1B: Add Redis Layer (2-3 days)

### Redis Integration
**Dependencies: Phase 1A complete**

**1B.1 Redis Configuration & Connection**
- Add Redis client to shared package with connection pooling
- Create Redis health checks and monitoring
- Implement connection retry logic and circuit breaker
- Add Redis configuration management (cluster vs single instance)

**1B.2 Session Management with Redis**
- Migrate session storage from in-memory to Redis
- Implement session middleware with Redis backend
- Add session invalidation and cleanup routines
- Create session security (encryption, tampering protection)

**1B.3 Caching Layer Implementation**
- Create cache abstraction layer for future use
- Add caching for frequently accessed data (questions, leaderboards)
- Implement cache invalidation strategies
- Add cache performance metrics

**1B.4 Update Development Environment**
- Extend `docker-compose.basic.yml` to include Redis
- Create `docker-compose.redis.yml` configuration
- Update health check scripts to verify Redis connectivity
- Add Redis monitoring and debugging tools

**1B.5 Redis Integration Testing**
- Extend integration tests to include Redis functionality
- Test session persistence across service restarts
- Validate cache behavior and invalidation
- Test Redis failover scenarios
- Add performance testing for cache operations

**Success Criteria for 1B:**
- Redis connects reliably with proper failover
- Sessions persist correctly in Redis
- Cache operations perform as expected
- Integration tests pass with Redis + PostgreSQL
- Services handle Redis connectivity issues gracefully

---

## Phase 1C: Add Zitadel Authentication (3-4 days)

### External Authentication Integration
**Dependencies: Phase 1B complete**

**1C.1 Zitadel Repository Implementation**
- Implement full ZitadelRepository interface with HTTP client
- Add JWT token validation with Zitadel public keys
- Create token refresh and user management functionality
- Implement circuit breaker for external API calls
- Add proper error handling and retry logic

**1C.2 Replace Mock Authentication**
- Migrate from mock JWT to real Zitadel integration
- Update middleware to validate tokens against Zitadel
- Implement proper role and permission mapping
- Add MFA validation for admin endpoints
- Create user context enrichment from Zitadel claims

**1C.3 Complete Docker Environment**
- Add Zitadel container to Docker Compose setup
- Create `docker-compose.yml` with full infrastructure stack
- Configure Zitadel with proper OAuth applications
- Set up development user accounts and roles
- Add monitoring for all services

**1C.4 Observability Foundation**
- Add Prometheus metrics collection to all services
- Create basic Grafana dashboards
- Implement distributed tracing preparation
- Add structured logging across services
- Create alerting for critical failures

**1C.5 End-to-End Integration Testing**
- Create full authentication flow tests
- Test token validation and refresh cycles
- Validate role-based access control
- Test multi-service communication with real auth
- Add performance testing under load

**Success Criteria for 1C:**
- Zitadel authentication works end-to-end
- All services integrate properly with real JWT validation
- Role-based access control functions correctly
- Integration tests pass with full infrastructure stack
- System handles authentication failures gracefully
- Basic observability provides useful insights

---

## Progressive Testing Strategy

### Phase 1A Testing Focus
- Database operations and data consistency
- Service startup and health check reliability
- Mock authentication token flows
- Basic API functionality

### Phase 1B Testing Focus
- Redis connectivity and session persistence
- Cache performance and invalidation
- Service behavior with Redis failures
- Session security and tampering protection

### Phase 1C Testing Focus
- External authentication integration
- Token validation and refresh flows
- Role-based access across services
- System resilience under auth failures
- End-to-end user workflows

## Benefits of Progressive Approach

1. **Incremental Complexity**: Each phase adds one major component
2. **Isolation of Issues**: Problems can be traced to specific layers
3. **Thorough Testing**: Each layer gets comprehensive validation
4. **Rollback Capability**: Can revert to previous working state
5. **Learning Curve**: Understand each technology deeply before combining
6. **Debugging Ease**: Fewer variables when troubleshooting
7. **Confidence Building**: Success at each phase builds momentum

## Deliverables Timeline

- **Week 1**: Phase 1A - Backend + Database foundation
- **Week 2**: Phase 1B - Redis integration and testing
- **Week 2-3**: Phase 1C - Zitadel integration and full stack testing

This progressive approach ensures solid foundations at each layer before adding the next level of complexity, making the overall implementation more reliable and maintainable.

---

## Detailed Implementation Steps for Phase 1A

### Step 1A.1: Create Go Project Structure

#### Root Go Module Setup
```bash
# Initialize main go.mod
go mod init knowfoolery
go mod tidy
```

#### Directory Structure Creation
```
knowfoolery/
├── backend/
│   ├── services/
│   │   ├── game-service/
│   │   │   ├── cmd/main.go
│   │   │   ├── internal/
│   │   │   │   ├── handlers/
│   │   │   │   ├── services/
│   │   │   │   ├── middleware/
│   │   │   │   └── models/
│   │   │   ├── config/
│   │   │   ├── tests/
│   │   │   └── go.mod
│   │   ├── question-service/
│   │   ├── user-service/
│   │   ├── leaderboard-service/
│   │   ├── session-service/
│   │   ├── admin-service/
│   │   └── gateway-service/
│   ├── shared/
│   │   ├── auth/
│   │   ├── database/
│   │   ├── observability/
│   │   ├── security/
│   │   └── utils/
│   └── scripts/
└── infrastructure/
    └── docker/
```

### Step 1A.2: Define Ent Schemas

#### Question Schema
```go
// backend/shared/database/ent/schema/question.go
package schema

import (
    "time"
    "entgo.io/ent"
    "entgo.io/ent/schema/field"
    "entgo.io/ent/schema/index"
)

type Question struct {
    ent.Schema
}

func (Question) Fields() []ent.Field {
    return []ent.Field{
        field.String("id").Unique(),
        field.String("theme").NotEmpty(),
        field.Text("text").NotEmpty(),
        field.String("answer").NotEmpty(),
        field.Text("hint").Optional(),
        field.Enum("difficulty").Values("easy", "medium", "hard").Default("medium"),
        field.Bool("is_active").Default(true),
        field.Time("created_at").Default(time.Now),
        field.Time("updated_at").Default(time.Now).UpdateDefault(time.Now),
    }
}

func (Question) Indexes() []ent.Index {
    return []ent.Index{
        index.Fields("theme"),
        index.Fields("difficulty"),
        index.Fields("is_active"),
        index.Fields("created_at"),
    }
}
```

### Step 1A.3: Basic Service Implementation

#### Game Service Main
```go
// backend/services/game-service/cmd/main.go
package main

import (
    "log"
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/cors"
    "github.com/gofiber/fiber/v2/middleware/logger"
)

func main() {
    app := fiber.New(fiber.Config{
        AppName: "Know Foolery Game Service",
    })

    // Middleware
    app.Use(logger.New())
    app.Use(cors.New())

    // Health endpoint
    app.Get("/health", func(c *fiber.Ctx) error {
        return c.JSON(fiber.Map{
            "status": "healthy",
            "service": "game-service",
        })
    })

    log.Println("Game Service starting on :3001")
    log.Fatal(app.Listen(":3001"))
}
```

### Step 1A.4: Docker Compose Basic Setup

#### Basic Docker Compose
```yaml
# infrastructure/docker/docker-compose.basic.yml
version: '3.8'

services:
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: knowfoolery
      POSTGRES_USER: knowfoolery
      POSTGRES_PASSWORD: dev-password
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-db.sql:/docker-entrypoint-initdb.d/init-db.sql
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U knowfoolery"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  postgres_data:
```

### Step 1A.5: Integration Test Framework

#### Test Utilities
```go
// backend/shared/database/testutil/testutil.go
package testutil

import (
    "context"
    "testing"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/modules/postgres"
)

func SetupTestDB(t *testing.T) (*ent.Client, func()) {
    ctx := context.Background()
    
    postgresContainer, err := postgres.RunContainer(ctx,
        testcontainers.WithImage("postgres:15-alpine"),
        postgres.WithDatabase("testdb"),
        postgres.WithUsername("test"),
        postgres.WithPassword("test"),
    )
    if err != nil {
        t.Fatal(err)
    }

    // Get connection string and create Ent client
    // ... implementation details

    cleanup := func() {
        postgresContainer.Terminate(ctx)
    }

    return client, cleanup
}
```

## Current Phase 1A Implementation Checklist

- [ ] Create complete Go project structure
- [ ] Initialize all service modules with basic Fiber apps
- [ ] Define and generate all Ent schemas
- [ ] Set up database connection and migration utilities
- [ ] Implement mock JWT authentication
- [ ] Create Docker Compose basic environment
- [ ] Set up integration testing framework
- [ ] Add health checks and basic monitoring
- [ ] Create database seeding scripts
- [ ] Validate all services start and communicate properly

This detailed plan provides clear, actionable steps for implementing Phase 1A while maintaining the progressive approach that allows for thorough testing at each layer.