107 lines
3.5 KiB
Markdown
107 lines
3.5 KiB
Markdown
# Database Migration Issues - SOLVED
|
|
|
|
## Problem Summary
|
|
You were experiencing unwanted tables being created and duplicates when starting the server. This was caused by multiple migration sources creating the same tables and conflicting migration execution.
|
|
|
|
## Root Causes Identified
|
|
|
|
### 1. **Multiple Migration Sources**
|
|
- PostgreSQL init script (`databases/scripts/init.sql`) creates the `dev_pipeline` database
|
|
- Shared schemas (`databases/scripts/schemas.sql`) creates core tables
|
|
- Individual service migrations create their own tables
|
|
- Template-manager was also applying shared schemas, causing duplicates
|
|
|
|
### 2. **Migration Execution Order Issues**
|
|
- Services were running migrations in parallel
|
|
- No proper dependency management between shared schemas and service-specific tables
|
|
- DROP TABLE statements in development mode causing data loss
|
|
|
|
### 3. **Table Conflicts**
|
|
- `users` table created by both `schemas.sql` and `user-auth` migration
|
|
- `user_projects` table created by both sources
|
|
- Function conflicts (`update_updated_at_column()` created multiple times)
|
|
- Extension conflicts (`uuid-ossp` created multiple times)
|
|
|
|
## Solutions Implemented
|
|
|
|
### 1. **Fixed Migration Order**
|
|
- Created separate `shared-schemas` service for core database tables
|
|
- Updated migration script to run in correct order:
|
|
1. `shared-schemas` (core tables first)
|
|
2. `user-auth` (user-specific tables)
|
|
3. `template-manager` (template-specific tables)
|
|
|
|
### 2. **Made Migrations Production-Safe**
|
|
- Replaced `DROP TABLE IF EXISTS` with `CREATE TABLE IF NOT EXISTS`
|
|
- Prevents data loss on server restarts
|
|
- Safe for production environments
|
|
|
|
### 3. **Eliminated Duplicate Table Creation**
|
|
- Removed shared schema application from template-manager
|
|
- Each service now only creates its own tables
|
|
- Proper dependency management
|
|
|
|
### 4. **Created Database Cleanup Script**
|
|
- `scripts/cleanup-database.sh` removes unwanted/duplicate tables
|
|
- Can be run to clean up existing database issues
|
|
|
|
## How to Use
|
|
|
|
### Clean Up Existing Database
|
|
```bash
|
|
cd /home/tech4biz/Desktop/Projectsnew/CODENUK1/codenuk-backend-live
|
|
./scripts/cleanup-database.sh
|
|
```
|
|
|
|
### Start Server with Fixed Migrations
|
|
```bash
|
|
docker-compose up --build
|
|
```
|
|
|
|
The migrations will now run in the correct order:
|
|
1. Shared schemas (projects, tech_stack_decisions, etc.)
|
|
2. User authentication tables
|
|
3. Template management tables
|
|
|
|
## Files Modified
|
|
|
|
1. **`services/template-manager/src/migrations/migrate.js`**
|
|
- Removed shared schema application
|
|
- Now only handles template-specific tables
|
|
|
|
2. **`services/user-auth/src/migrations/001_user_auth_schema.sql`**
|
|
- Replaced DROP TABLE with CREATE TABLE IF NOT EXISTS
|
|
- Made migration production-safe
|
|
|
|
3. **`services/template-manager/src/migrations/001_initial_schema.sql`**
|
|
- Replaced DROP TABLE with CREATE TABLE IF NOT EXISTS
|
|
- Made migration production-safe
|
|
|
|
4. **`scripts/migrate-all.sh`**
|
|
- Added shared-schemas service
|
|
- Proper migration order
|
|
|
|
5. **`docker-compose.yml`**
|
|
- Removed APPLY_SCHEMAS_SQL environment variable
|
|
|
|
6. **Created new files:**
|
|
- `services/shared-schemas/` - Dedicated service for shared schemas
|
|
- `scripts/cleanup-database.sh` - Database cleanup script
|
|
|
|
## Expected Results
|
|
|
|
After these changes:
|
|
- ✅ No duplicate tables will be created
|
|
- ✅ No unwanted tables from pgAdmin
|
|
- ✅ Proper migration order
|
|
- ✅ Production-safe migrations
|
|
- ✅ Clean database schema
|
|
|
|
## Verification
|
|
|
|
To verify the fix worked:
|
|
1. Run the cleanup script
|
|
2. Start the server
|
|
3. Check pgAdmin - you should only see the intended tables
|
|
4. No duplicate or unwanted tables should appear
|