# Database Migration System - Clean & Organized

## Overview

This document explains the new clean database migration system that resolves the issues with unwanted tables and duplicate table creation.

## Problems Solved

### ❌ Previous Issues
- **Duplicate tables**: Multiple services creating the same tables (`users`, `user_projects`, etc.)
- **Unwanted tables**: Tech-stack-selector creating massive schema with 100+ tables
- **Inconsistent migrations**: Some services using `DROP TABLE`, others using `CREATE TABLE IF NOT EXISTS`
- **Missing shared-schemas**: Migration script referenced non-existent service
- **AI-mockup-service duplication**: Creating same tables as user-auth service

### ✅ Solutions Implemented

1. **Clean Database Reset**: Complete schema reset before applying migrations
2. **Proper Migration Order**: Core schema first, then service-specific tables
3. **Minimal Service Schemas**: Each service only creates tables it actually needs
4. **Consistent Approach**: All services use `CREATE TABLE IF NOT EXISTS`
5. **Migration Tracking**: Proper tracking of applied migrations

## Migration System Architecture

### 1. Core Schema (databases/scripts/schemas.sql)
**Tables Created:**
- `projects` - Main project tracking
- `tech_stack_decisions` - Technology choices per project
- `system_architectures` - Architecture designs
- `code_generations` - Generated code tracking
- `test_results` - Test execution results
- `deployment_logs` - Deployment tracking
- `service_health` - Service monitoring
- `project_state_transitions` - Audit trail

### 2. Service-Specific Tables

#### User Authentication Service (`user-auth`)
**Tables Created:**
- `users` - User accounts
- `refresh_tokens` - JWT refresh tokens
- `user_sessions` - User session tracking
- `user_feature_preferences` - Feature customization
- `user_projects` - User project tracking

#### Template Manager Service (`template-manager`)
**Tables Created:**
- `templates` - Template definitions
- `template_features` - Feature definitions
- `feature_usage` - Usage tracking
- `custom_features` - User-created features

#### Requirement Processor Service (`requirement-processor`)
**Tables Created:**
- `business_context_responses` - Business context data
- `question_templates` - Reusable question sets

#### Git Integration Service (`git-integration`)
**Tables Created:**
- `github_repositories` - Repository tracking
- `github_user_tokens` - OAuth tokens
- `repository_storage` - Local storage tracking
- `repository_directories` - Directory structure
- `repository_files` - File tracking

#### AI Mockup Service (`ai-mockup-service`)
**Tables Created:**
- `wireframes` - Wireframe data
- `wireframe_versions` - Version tracking
- `wireframe_elements` - Element analysis

#### Tech Stack Selector Service (`tech-stack-selector`)
**Tables Created:**
- `tech_stack_recommendations` - AI recommendations
- `stack_analysis_cache` - Analysis caching

## How to Use

### Clean Database Migration

```bash
cd /home/tech4biz/Desktop/Projectsnew/CODENUK1/codenuk-backend-live

# Run the clean migration script
./scripts/migrate-clean.sh
```

### Start Services with Clean Database

```bash
# Start all services with clean migrations
docker-compose up --build

# Or start specific services
docker-compose up postgres redis migrations
```

### Manual Database Cleanup (if needed)

```bash
# Run the cleanup script to remove unwanted tables
./scripts/cleanup-database.sh
```

## Migration Process

### Step 1: Database Cleanup
- Drops all existing tables
- Recreates public schema
- Re-enables required extensions
- Creates migration tracking table

### Step 2: Core Schema Application
- Applies `databases/scripts/schemas.sql`
- Creates core pipeline tables
- Marks as applied in migration tracking

### Step 3: Service Migrations
- Runs migrations in dependency order:
  1. `user-auth` (user tables first)
  2. `template-manager` (template tables)
  3. `requirement-processor` (business context)
  4. `git-integration` (repository tracking)
  5. `ai-mockup-service` (wireframe tables)
  6. `tech-stack-selector` (recommendation tables)

### Step 4: Verification
- Lists all created tables
- Shows applied migrations
- Confirms successful completion

## Service Migration Scripts

### Node.js Services
- `user-auth`: `npm run migrate`
- `template-manager`: `npm run migrate`
- `git-integration`: `npm run migrate`

### Python Services
- `ai-mockup-service`: `python3 src/migrations/migrate.py`
- `tech-stack-selector`: `python3 migrate.py`
- `requirement-processor`: `python3 migrations/migrate.py`

## Expected Final Tables

After running the clean migration, you should see these tables:

### Core Tables (8)
- `projects`
- `tech_stack_decisions`
- `system_architectures`
- `code_generations`
- `test_results`
- `deployment_logs`
- `service_health`
- `project_state_transitions`

### User Auth Tables (5)
- `users`
- `refresh_tokens`
- `user_sessions`
- `user_feature_preferences`
- `user_projects`

### Template Manager Tables (4)
- `templates`
- `template_features`
- `feature_usage`
- `custom_features`

### Requirement Processor Tables (2)
- `business_context_responses`
- `question_templates`

### Git Integration Tables (5)
- `github_repositories`
- `github_user_tokens`
- `repository_storage`
- `repository_directories`
- `repository_files`

### AI Mockup Tables (3)
- `wireframes`
- `wireframe_versions`
- `wireframe_elements`

### Tech Stack Selector Tables (2)
- `tech_stack_recommendations`
- `stack_analysis_cache`

### System Tables (1)
- `schema_migrations`

**Total: 29 tables** (vs 100+ previously)

## Troubleshooting

### If Migration Fails
1. Check database connection parameters
2. Ensure all required extensions are available
3. Verify service directories exist
4. Check migration script permissions

### If Unwanted Tables Appear
1. Run `./scripts/cleanup-database.sh`
2. Restart with `docker-compose up --build`
3. Check service migration scripts for DROP statements

### If Services Don't Start
1. Check migration dependencies in docker-compose.yml
2. Verify migration script completed successfully
3. Check service logs for database connection issues

## Benefits

✅ **Clean Database**: Only necessary tables created
✅ **No Duplicates**: Each table created by one service only
✅ **Proper Dependencies**: Tables created in correct order
✅ **Production Safe**: Uses `CREATE TABLE IF NOT EXISTS`
✅ **Trackable**: All migrations tracked and logged
✅ **Maintainable**: Clear separation of concerns
✅ **Scalable**: Easy to add new services

## Next Steps

1. **Test the migration**: Run `./scripts/migrate-clean.sh`
2. **Start services**: Run `docker-compose up --build`
3. **Verify tables**: Check pgAdmin for clean table list
4. **Monitor logs**: Ensure all services start successfully

The database is now clean, organized, and ready for production use!