# Database Migration Issues - SOLVED

## Problem Summary
You were experiencing unwanted tables being created and duplicates when starting the server. This was caused by multiple migration sources creating the same tables and conflicting migration execution.

## Root Causes Identified

### 1. **Multiple Migration Sources**
- PostgreSQL init script (`databases/scripts/init.sql`) creates the `dev_pipeline` database
- Shared schemas (`databases/scripts/schemas.sql`) creates core tables
- Individual service migrations create their own tables
- Template-manager was also applying shared schemas, causing duplicates

### 2. **Migration Execution Order Issues**
- Services were running migrations in parallel
- No proper dependency management between shared schemas and service-specific tables
- DROP TABLE statements in development mode causing data loss

### 3. **Table Conflicts**
- `users` table created by both `schemas.sql` and `user-auth` migration
- `user_projects` table created by both sources
- Function conflicts (`update_updated_at_column()` created multiple times)
- Extension conflicts (`uuid-ossp` created multiple times)

## Solutions Implemented

### 1. **Fixed Migration Order**
- Created separate `shared-schemas` service for core database tables
- Updated migration script to run in correct order:
  1. `shared-schemas` (core tables first)
  2. `user-auth` (user-specific tables)
  3. `template-manager` (template-specific tables)

### 2. **Made Migrations Production-Safe**
- Replaced `DROP TABLE IF EXISTS` with `CREATE TABLE IF NOT EXISTS`
- Prevents data loss on server restarts
- Safe for production environments

### 3. **Eliminated Duplicate Table Creation**
- Removed shared schema application from template-manager
- Each service now only creates its own tables
- Proper dependency management

### 4. **Created Database Cleanup Script**
- `scripts/cleanup-database.sh` removes unwanted/duplicate tables
- Can be run to clean up existing database issues

## How to Use

### Clean Up Existing Database
```bash
cd /home/tech4biz/Desktop/Projectsnew/CODENUK1/codenuk-backend-live
./scripts/cleanup-database.sh
```

### Start Server with Fixed Migrations
```bash
docker-compose up --build
```

The migrations will now run in the correct order:
1. Shared schemas (projects, tech_stack_decisions, etc.)
2. User authentication tables
3. Template management tables

## Files Modified

1. **`services/template-manager/src/migrations/migrate.js`**
   - Removed shared schema application
   - Now only handles template-specific tables

2. **`services/user-auth/src/migrations/001_user_auth_schema.sql`**
   - Replaced DROP TABLE with CREATE TABLE IF NOT EXISTS
   - Made migration production-safe

3. **`services/template-manager/src/migrations/001_initial_schema.sql`**
   - Replaced DROP TABLE with CREATE TABLE IF NOT EXISTS
   - Made migration production-safe

4. **`scripts/migrate-all.sh`**
   - Added shared-schemas service
   - Proper migration order

5. **`docker-compose.yml`**
   - Removed APPLY_SCHEMAS_SQL environment variable

6. **Created new files:**
   - `services/shared-schemas/` - Dedicated service for shared schemas
   - `scripts/cleanup-database.sh` - Database cleanup script

## Expected Results

After these changes:
- ✅ No duplicate tables will be created
- ✅ No unwanted tables from pgAdmin
- ✅ Proper migration order
- ✅ Production-safe migrations
- ✅ Clean database schema

## Verification

To verify the fix worked:
1. Run the cleanup script
2. Start the server
3. Check pgAdmin - you should only see the intended tables
4. No duplicate or unwanted tables should appear