prakash/codenuk_backend_mine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

backend changes

Browse Source
This commit is contained in:
Chandini 2025-10-02 12:13:20 +05:30
parent 2f9c61bcc9
commit 7285c25aac
49 changed files with 5446 additions and 557 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

171
DEPLOYMENT_FIX_GUIDE.md Normal file
View File

@ -0,0 +1,171 @@
# 🚀 Microservices Deployment Fix Guide
## 🔍 Issues Identified
### 1. N8N Service Failure (Exit Code 1)
- **Root Cause**: Database schema conflicts and timing issues
- **Symptoms**: `pipeline_n8n` container exits with code 1
### 2. PostgreSQL Constraint Violations
- **Root Cause**: Duplicate type/table creation attempts
- **Error**: `duplicate key value violates unique constraint "pg_type_typname_nsp_index"`
## 🛠️ Solutions Implemented
### 1. Enhanced N8N Configuration
- Added dedicated `n8n` schema
- Improved health checks with longer start period
- Added restart policy and better logging
- Ensured proper dependency ordering
### 2. Database Schema Cleanup
- Created schema conflict resolution script
- Proper table ownership and permissions
- Separated n8n tables into dedicated schema
### 3. Deployment Orchestration
- Staged service startup to prevent race conditions
- Proper dependency management
- Volume cleanup for fresh starts
## 🚀 Deployment Steps
### Option 1: Automated Fix (Recommended)
```bash
cd /home/tech4biz/Desktop/Projectsnew/CODENUK1/codenuk-backend-live
./scripts/fix-deployment-issues.sh
```
### Option 2: Manual Step-by-Step
#### Step 1: Clean Environment
```bash
# Stop all services
docker-compose down --volumes --remove-orphans
# Clean Docker system
docker system prune -f
docker volume prune -f
# Remove problematic volumes
docker volume rm codenuk-backend-live_postgres_data 2>/dev/null || true
docker volume rm codenuk-backend-live_n8n_data 2>/dev/null || true
```
#### Step 2: Start Core Infrastructure
```bash
# Start databases first
docker-compose up -d postgres redis mongodb rabbitmq
# Wait for readiness
sleep 30
```
#### Step 3: Fix Database Schema
```bash
# Apply schema fixes
docker exec -i pipeline_postgres psql -U pipeline_admin -d dev_pipeline < databases/scripts/fix-schema-conflicts.sql
```
#### Step 4: Run Migrations
```bash
docker-compose up migrations
```
#### Step 5: Start Services in Stages
```bash
# Stage 1: Core services
docker-compose up -d n8n api-gateway requirement-processor
# Stage 2: Generation services
docker-compose up -d tech-stack-selector architecture-designer code-generator
# Stage 3: User services
docker-compose up -d user-auth template-manager
# Stage 4: Additional services
docker-compose up -d ai-mockup-service git-integration web-dashboard
```
## 🏥 Health Verification
### Check Service Status
```bash
docker-compose ps
```
### Check N8N Specifically
```bash
# Check n8n logs
docker-compose logs n8n
# Test n8n endpoint
curl -f http://localhost:5678/healthz
```
### Check Database
```bash
# Connect to database
docker exec -it pipeline_postgres psql -U pipeline_admin -d dev_pipeline
# List schemas
\dn
# Check n8n tables
\dt n8n.*
```
## 🔧 Troubleshooting
### If N8N Still Fails
1. Check logs: `docker-compose logs n8n`
2. Verify database connection: `docker exec pipeline_postgres pg_isready`
3. Check n8n schema exists: `docker exec -it pipeline_postgres psql -U pipeline_admin -d dev_pipeline -c "\dn"`
### If Database Conflicts Persist
1. Run schema cleanup again: `docker exec -i pipeline_postgres psql -U pipeline_admin -d dev_pipeline < databases/scripts/fix-schema-conflicts.sql`
2. Check for remaining conflicts: `docker-compose logs postgres | grep ERROR`
### If Services Won't Start
1. Check dependencies: `docker-compose config --services`
2. Start services individually: `docker-compose up [service-name]`
3. Check resource usage: `docker stats`
## 📊 Expected Results
After successful deployment:
- ✅ All services should show "Up" status
- ✅ N8N accessible at http://localhost:5678
- ✅ No database constraint errors
- ✅ All health checks passing
## 🎯 Key Improvements Made
1. **N8N Configuration**:
- Dedicated schema isolation
- Better dependency management
- Improved health checks
- Restart policies
2. **Database Management**:
- Schema conflict resolution
- Proper table ownership
- Clean migration process
3. **Deployment Process**:
- Staged service startup
- Volume cleanup
- Dependency ordering
## 📞 Support
If issues persist:
1. Check service logs: `docker-compose logs [service-name]`
2. Verify network connectivity: `docker network ls`
3. Check resource usage: `docker system df`
4. Review configuration: `docker-compose config`
---
**Last Updated**: September 30, 2025
**Build Number**: 24+ (Fixed)
**Status**: ✅ Ready for Deployment

64
databases/scripts/fix-schema-conflicts.sql Normal file
View File

@ -0,0 +1,64 @@
-- Fix Schema Conflicts and Prepare for Clean Deployment
-- This script resolves duplicate key constraint violations
-- Create n8n schema if it doesn't exist
CREATE SCHEMA IF NOT EXISTS n8n;
-- Clean up any existing conflicting types/tables
DROP TYPE IF EXISTS claude_recommendations CASCADE;
DROP TABLE IF EXISTS claude_recommendations CASCADE;
-- Clean up n8n related tables if they exist in public schema
DROP TABLE IF EXISTS public.n8n_credentials_entity CASCADE;
DROP TABLE IF EXISTS public.n8n_execution_entity CASCADE;
DROP TABLE IF EXISTS public.n8n_workflow_entity CASCADE;
DROP TABLE IF EXISTS public.n8n_webhook_entity CASCADE;
DROP TABLE IF EXISTS public.n8n_tag_entity CASCADE;
DROP TABLE IF EXISTS public.n8n_workflows_tags CASCADE;
-- Reset any conflicting sequences
DROP SEQUENCE IF EXISTS claude_recommendations_id_seq CASCADE;
-- Ensure proper permissions for n8n schema
GRANT ALL PRIVILEGES ON SCHEMA n8n TO pipeline_admin;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA n8n TO pipeline_admin;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA n8n TO pipeline_admin;
-- Create claude_recommendations table properly (if needed by services)
CREATE TABLE IF NOT EXISTS claude_recommendations (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
request_id VARCHAR(255) UNIQUE NOT NULL,
domain VARCHAR(100) NOT NULL,
budget DECIMAL(10,2) NOT NULL,
preferred_technologies TEXT[],
template_id UUID,
stack_name VARCHAR(255) NOT NULL,
monthly_cost DECIMAL(10,2) NOT NULL,
setup_cost DECIMAL(10,2) NOT NULL,
team_size VARCHAR(50) NOT NULL,
development_time INTEGER NOT NULL,
satisfaction INTEGER NOT NULL CHECK (satisfaction >= 0 AND satisfaction <= 100),
success_rate INTEGER NOT NULL CHECK (success_rate >= 0 AND success_rate <= 100),
frontend VARCHAR(100) NOT NULL,
backend VARCHAR(100) NOT NULL,
database VARCHAR(100) NOT NULL,
cloud VARCHAR(100) NOT NULL,
testing VARCHAR(100) NOT NULL,
mobile VARCHAR(100),
devops VARCHAR(100) NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
-- Create indexes for better performance
CREATE INDEX IF NOT EXISTS idx_claude_recommendations_request_id ON claude_recommendations(request_id);
CREATE INDEX IF NOT EXISTS idx_claude_recommendations_domain ON claude_recommendations(domain);
CREATE INDEX IF NOT EXISTS idx_claude_recommendations_created_at ON claude_recommendations(created_at);
-- Ensure proper ownership
ALTER TABLE claude_recommendations OWNER TO pipeline_admin;
-- Log the cleanup
INSERT INTO schema_migrations (service, version, description, applied_at)
VALUES ('database-cleanup', 'fix_schema_conflicts', 'Fixed schema conflicts and prepared for clean deployment', CURRENT_TIMESTAMP)
ON CONFLICT (service, version) DO NOTHING;

14
docker-compose.yml
View File

@ -703,7 +703,7 @@ services:
- HOST=0.0.0.0
- DATABASE_URL=postgresql://pipeline_admin:secure_pipeline_2024@postgres:5432/dev_pipeline
- CLAUDE_API_KEY=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
- REDIS_URL=redis://:redis_secure_2024@pipeline_redis:6379
- REDIS_URL=redis://:redis_secure_2024@redis:6379
- SERVICE_PORT=8007
- LOG_LEVEL=INFO
- DEFAULT_TARGET_QUALITY=0.85
@ -726,9 +726,9 @@ services:
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8007/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
timeout: 15s
retries: 5
start_period: 120s
# =====================================
# Workflow Orchestration
# =====================================
@ -795,6 +795,9 @@ services:
- DB_POSTGRESDB_DATABASE=dev_pipeline
- DB_POSTGRESDB_USER=pipeline_admin
- DB_POSTGRESDB_PASSWORD=secure_pipeline_2024
- DB_POSTGRESDB_SCHEMA=n8n
- N8N_LOG_LEVEL=info
- N8N_METRICS=true
volumes:
- n8n_data:/home/node/.n8n
- ./orchestration/n8n/workflows:/home/node/.n8n/workflows
@ -807,12 +810,15 @@ services:
condition: service_healthy
rabbitmq:
condition: service_healthy
migrations:
condition: service_completed_successfully
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:5678/healthz"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
restart: unless-stopped
# =====================================
# Volumes

156
scripts/fix-deployment-issues.sh Executable file
View File

@ -0,0 +1,156 @@
#!/bin/bash
echo "🔧 Fixing Microservices Deployment Issues"
echo "========================================"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Function to print colored output
print_status() {
echo -e "${GREEN}[INFO]${NC} $1"
}
print_warning() {
echo -e "${YELLOW}[WARNING]${NC} $1"
}
print_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
# 1. Stop all services
print_status "Stopping all services..."
docker-compose down --volumes --remove-orphans
# 2. Clean up Docker system
print_status "Cleaning up Docker system..."
docker system prune -f
docker volume prune -f
# 3. Remove problematic volumes
print_status "Removing problematic volumes..."
docker volume rm codenuk-backend-live_postgres_data 2>/dev/null || true
docker volume rm codenuk-backend-live_n8n_data 2>/dev/null || true
docker volume rm codenuk-backend-live_migration_state 2>/dev/null || true
# 4. Clean database schema conflicts
print_status "Preparing clean database environment..."
cat > /tmp/clean_db.sql << 'EOF'
-- Clean up any existing schema conflicts
DROP TYPE IF EXISTS claude_recommendations CASCADE;
DROP TABLE IF EXISTS claude_recommendations CASCADE;
-- Clean up n8n related tables if they exist
DROP TABLE IF EXISTS n8n_credentials_entity CASCADE;
DROP TABLE IF EXISTS n8n_execution_entity CASCADE;
DROP TABLE IF EXISTS n8n_workflow_entity CASCADE;
DROP TABLE IF EXISTS n8n_webhook_entity CASCADE;
DROP TABLE IF EXISTS n8n_tag_entity CASCADE;
DROP TABLE IF EXISTS n8n_workflows_tags CASCADE;
-- Reset any conflicting sequences
DROP SEQUENCE IF EXISTS claude_recommendations_id_seq CASCADE;
EOF
# 5. Start only core infrastructure first
print_status "Starting core infrastructure services..."
docker-compose up -d postgres redis mongodb rabbitmq
# 6. Wait for databases to be ready
print_status "Waiting for databases to be ready..."
sleep 30
# Check if postgres is ready
print_status "Checking PostgreSQL readiness..."
for i in {1..30}; do
if docker exec pipeline_postgres pg_isready -U pipeline_admin -d dev_pipeline; then
print_status "PostgreSQL is ready!"
break
fi
print_warning "Waiting for PostgreSQL... ($i/30)"
sleep 2
done
# 7. Clean the database
print_status "Cleaning database schema conflicts..."
docker exec -i pipeline_postgres psql -U pipeline_admin -d dev_pipeline < /tmp/clean_db.sql 2>/dev/null || true
# 8. Run migrations
print_status "Running database migrations..."
docker-compose up migrations
# Wait for migrations to complete
print_status "Waiting for migrations to complete..."
sleep 10
# 9. Start n8n with proper initialization
print_status "Starting n8n service..."
docker-compose up -d n8n
# Wait for n8n to initialize
print_status "Waiting for n8n to initialize..."
sleep 20
# 10. Start remaining services in batches
print_status "Starting core services..."
docker-compose up -d \
api-gateway \
requirement-processor \
tech-stack-selector \
architecture-designer
sleep 15
print_status "Starting generation services..."
docker-compose up -d \
code-generator \
test-generator \
deployment-manager
sleep 15
print_status "Starting user services..."
docker-compose up -d \
user-auth \
template-manager \
unison
sleep 15
print_status "Starting additional services..."
docker-compose up -d \
ai-mockup-service \
git-integration \
self-improving-generator \
web-dashboard
# 11. Final health check
print_status "Performing final health check..."
sleep 30
echo ""
echo "🏥 Service Health Check"
echo "======================"
# Check service status
docker-compose ps
echo ""
print_status "Deployment fix completed!"
print_warning "Please check the service status above."
print_warning "If any services are still failing, check their logs with:"
print_warning "docker-compose logs [service-name]"
# Clean up temp file
rm -f /tmp/clean_db.sql
echo ""
print_status "🎯 Next Steps:"
echo "1. Check service logs: docker-compose logs -f"
echo "2. Verify n8n is accessible: http://localhost:5678"
echo "3. Test API endpoints for health checks"
echo "4. Monitor for any remaining issues"

2
self-improving-generator/Dockerfile
View File

@ -24,7 +24,7 @@ USER app
EXPOSE 8007
# Health check
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
HEALTHCHECK --interval=30s --timeout=15s --start-period=120s --retries=5 \
CMD curl -f http://localhost:8007/health || exit 1
# Start the application

107
self-improving-generator/src/main.py
View File

@ -6,11 +6,13 @@ FastAPI application entry point for the self-improving code generator
import logging
import asyncio
import time
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, BackgroundTasks, Depends
from fastapi.middleware.cors import CORSMiddleware
from sqlalchemy import create_engine
from sqlalchemy import create_engine, text
from sqlalchemy.orm import sessionmaker
from sqlalchemy.exc import OperationalError
from .utils.config import get_settings, validate_configuration
from .models.database_models import Base
@ -29,6 +31,35 @@ generator: SelfImprovingCodeGenerator = None
engine = None
SessionLocal = None
async def wait_for_database(database_url: str, max_retries: int = 30, delay: float = 2.0):
"""Wait for database to be available with retry logic"""
for attempt in range(max_retries):
try:
logger.info(f"Attempting database connection (attempt {attempt + 1}/{max_retries})")
test_engine = create_engine(database_url)
# Test the connection
with test_engine.connect() as conn:
conn.execute(text("SELECT 1"))
logger.info("✅ Database connection successful")
test_engine.dispose()
return True
except OperationalError as e:
if attempt < max_retries - 1:
logger.warning(f"Database connection failed (attempt {attempt + 1}): {e}")
logger.info(f"Retrying in {delay} seconds...")
await asyncio.sleep(delay)
else:
logger.error(f"Failed to connect to database after {max_retries} attempts: {e}")
raise
except Exception as e:
logger.error(f"Unexpected error connecting to database: {e}")
raise
return False
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Application lifespan events"""
@ -41,6 +72,9 @@ async def lifespan(app: FastAPI):
logger.info("🚀 Starting Self-Improving Code Generator")
# Wait for database to be available
await wait_for_database(settings.database_url)
# Initialize database
engine = create_engine(settings.database_url)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
@ -63,6 +97,8 @@ async def lifespan(app: FastAPI):
raise
finally:
logger.info("🛑 Shutting down Self-Improving Code Generator")
if engine:
engine.dispose()
# Create FastAPI app
app = FastAPI(
@ -122,30 +158,53 @@ async def root():
@app.get("/health")
async def health_check():
"""Health check endpoint"""
settings = get_settings()
health_status = {
"status": "healthy",
"service": "Self-Improving Code Generator",
"version": "1.0.0",
"timestamp": "2024-01-01T00:00:00Z",
"dependencies": {
"database": "connected" if engine else "disconnected",
"claude_api": "configured" if settings.claude_api_key else "not_configured",
"generator": "initialized" if generator else "not_initialized"
try:
settings = get_settings()
# Test database connection
db_status = "disconnected"
if engine:
try:
with engine.connect() as conn:
conn.execute(text("SELECT 1"))
db_status = "connected"
except Exception as e:
logger.warning(f"Database health check failed: {e}")
db_status = "error"
health_status = {
"status": "healthy",
"service": "Self-Improving Code Generator",
"version": "1.0.0",
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"dependencies": {
"database": db_status,
"claude_api": "configured" if settings.claude_api_key else "not_configured",
"generator": "initialized" if generator else "not_initialized"
}
}
# Check if all dependencies are healthy
all_healthy = (
health_status["dependencies"]["database"] == "connected" and
health_status["dependencies"]["claude_api"] == "configured" and
health_status["dependencies"]["generator"] == "initialized"
)
if not all_healthy:
health_status["status"] = "unhealthy"
return health_status
except Exception as e:
logger.error(f"Health check failed: {e}")
return {
"status": "error",
"service": "Self-Improving Code Generator",
"version": "1.0.0",
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"error": str(e)
}
}
# Check if all dependencies are healthy
all_healthy = all(
status == "connected" or status == "configured" or status == "initialized"
for status in health_status["dependencies"].values()
)
if not all_healthy:
health_status["status"] = "unhealthy"
return health_status
if __name__ == "__main__":
import uvicorn

BIN
services/git-integration.zip
View File

Binary file not shown.

111
services/git-integration/package-lock.json generated
View File

@ -9,6 +9,7 @@
"version": "1.0.0",
"dependencies": {
"@octokit/rest": "^20.0.2",
"axios": "^1.12.2",
"cors": "^2.8.5",
"dotenv": "^16.3.1",
"express": "^4.18.2",
@ -211,6 +212,23 @@
"integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==",
"license": "MIT"
},
"node_modules/asynckit": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz",
"integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==",
"license": "MIT"
},
"node_modules/axios": {
"version": "1.12.2",
"resolved": "https://registry.npmjs.org/axios/-/axios-1.12.2.tgz",
"integrity": "sha512-vMJzPewAlRyOgxV2dU0Cuz2O8zzzx9VYtbJOaBgXFeLc4IV/Eg50n4LowmehOOR61S8ZMpc2K5Sa7g6A4jfkUw==",
"license": "MIT",
"dependencies": {
"follow-redirects": "^1.15.6",
"form-data": "^4.0.4",
"proxy-from-env": "^1.1.0"
}
},
"node_modules/balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
@ -366,6 +384,18 @@
"fsevents": "~2.3.2"
}
},
"node_modules/combined-stream": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz",
"integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==",
"license": "MIT",
"dependencies": {
"delayed-stream": "~1.0.0"
},
"engines": {
"node": ">= 0.8"
}
},
"node_modules/concat-map": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
@ -431,6 +461,15 @@
"ms": "2.0.0"
}
},
"node_modules/delayed-stream": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz",
"integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==",
"license": "MIT",
"engines": {
"node": ">=0.4.0"
}
},
"node_modules/depd": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
@ -527,6 +566,21 @@
"node": ">= 0.4"
}
},
"node_modules/es-set-tostringtag": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz",
"integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
"get-intrinsic": "^1.2.6",
"has-tostringtag": "^1.0.2",
"hasown": "^2.0.2"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/escape-html": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
@ -653,6 +707,42 @@
"node": ">= 0.8"
}
},
"node_modules/follow-redirects": {
"version": "1.15.11",
"resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.15.11.tgz",
"integrity": "sha512-deG2P0JfjrTxl50XGCDyfI97ZGVCxIpfKYmfyrQ54n5FO/0gfIES8C/Psl6kWVDolizcaaxZJnTS0QSMxvnsBQ==",
"funding": [
{
"type": "individual",
"url": "https://github.com/sponsors/RubenVerborgh"
}
],
"license": "MIT",
"engines": {
"node": ">=4.0"
},
"peerDependenciesMeta": {
"debug": {
"optional": true
}
}
},
"node_modules/form-data": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.4.tgz",
"integrity": "sha512-KrGhL9Q4zjj0kiUt5OO4Mr/A/jlI2jDYs5eHBpYHPcBEVSiipAvn2Ko2HnPe20rmcuuvMHNdZFp+4IlGTMF0Ow==",
"license": "MIT",
"dependencies": {
"asynckit": "^0.4.0",
"combined-stream": "^1.0.8",
"es-set-tostringtag": "^2.1.0",
"hasown": "^2.0.2",
"mime-types": "^2.1.12"
},
"engines": {
"node": ">= 6"
}
},
"node_modules/forwarded": {
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
@ -779,6 +869,21 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/has-tostringtag": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz",
"integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==",
"license": "MIT",
"dependencies": {
"has-symbols": "^1.0.3"
},
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/hasown": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
@ -1317,6 +1422,12 @@
"node": ">= 0.10"
}
},
"node_modules/proxy-from-env": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz",
"integrity": "sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg==",
"license": "MIT"
},
"node_modules/pstree.remy": {
"version": "1.1.8",
"resolved": "https://registry.npmjs.org/pstree.remy/-/pstree.remy-1.1.8.tgz",

1
services/git-integration/package.json
View File

@ -10,6 +10,7 @@
},
"dependencies": {
"@octokit/rest": "^20.0.2",
"axios": "^1.12.2",
"cors": "^2.8.5",
"dotenv": "^16.3.1",
"express": "^4.18.2",

63
services/git-integration/src/app.js
View File

@ -8,12 +8,20 @@ const morgan = require('morgan');
// Import database
const database = require('./config/database');
// Import services
const EnhancedDiffProcessingService = require('./services/enhanced-diff-processing.service');
// Import routes
const githubRoutes = require('./routes/github-integration.routes');
const githubOAuthRoutes = require('./routes/github-oauth');
const webhookRoutes = require('./routes/webhook.routes');
const vcsRoutes = require('./routes/vcs.routes');
// Import new enhanced routes
const commitsRoutes = require('./routes/commits.routes');
const oauthProvidersRoutes = require('./routes/oauth-providers.routes');
const enhancedWebhooksRoutes = require('./routes/enhanced-webhooks.routes');
const app = express();
const PORT = process.env.PORT || 8012;
@ -21,8 +29,21 @@ const PORT = process.env.PORT || 8012;
app.use(helmet());
app.use(cors());
app.use(morgan('combined'));
app.use(express.json({ limit: '10mb' }));
app.use(express.urlencoded({ extended: true }));
// Preserve raw body for webhook signature verification
app.use(express.json({
limit: '10mb',
verify: (req, res, buf) => {
req.rawBody = buf;
}
}));
app.use(express.urlencoded({
extended: true,
verify: (req, res, buf) => {
req.rawBody = buf;
}
}));
// Session middleware
app.use(session({
@ -42,6 +63,11 @@ app.use('/api/github', githubOAuthRoutes);
app.use('/api/github', webhookRoutes);
app.use('/api/vcs', vcsRoutes);
// Enhanced routes
app.use('/api/commits', commitsRoutes);
app.use('/api/oauth', oauthProvidersRoutes);
app.use('/api/webhooks', enhancedWebhooksRoutes);
// Health check endpoint
app.get('/health', (req, res) => {
res.status(200).json({
@ -63,7 +89,10 @@ app.get('/', (req, res) => {
github: '/api/github',
oauth: '/api/github/auth',
webhook: '/api/github/webhook',
vcs: '/api/vcs/:provider'
vcs: '/api/vcs/:provider',
commits: '/api/commits',
oauth_providers: '/api/oauth',
enhanced_webhooks: '/api/webhooks'
}
});
});
@ -99,11 +128,37 @@ process.on('SIGINT', async () => {
process.exit(0);
});
// Initialize services
async function initializeServices() {
try {
// Initialize diff processing service
const diffProcessingService = new EnhancedDiffProcessingService();
await diffProcessingService.initialize();
// Start background diff processing if enabled
if (process.env.ENABLE_BACKGROUND_DIFF_PROCESSING !== 'false') {
const processingInterval = parseInt(process.env.DIFF_PROCESSING_INTERVAL_MS) || 30000;
diffProcessingService.startBackgroundProcessing(processingInterval);
console.log('🔄 Background diff processing started');
}
console.log('✅ All services initialized successfully');
} catch (error) {
console.error('❌ Error initializing services:', error);
}
}
// Start server
app.listen(PORT, '0.0.0.0', () => {
app.listen(PORT, '0.0.0.0', async () => {
console.log(`🚀 Git Integration Service running on port ${PORT}`);
console.log(`📊 Health check: http://localhost:${PORT}/health`);
console.log(`🔗 GitHub API: http://localhost:${PORT}/api/github`);
console.log(`📝 Commits API: http://localhost:${PORT}/api/commits`);
console.log(`🔐 OAuth API: http://localhost:${PORT}/api/oauth`);
console.log(`🪝 Enhanced Webhooks: http://localhost:${PORT}/api/webhooks`);
// Initialize services after server starts
await initializeServices();
});
module.exports = app;

8
services/git-integration/src/migrations/001_github_integration.sql
View File

@ -2,7 +2,7 @@
-- This migration adds support for GitHub repository integration
-- Create table for GitHub repositories
CREATE TABLE IF NOT EXISTS github_repositories (
CREATE TABLE IF NOT EXISTS "github_repositories@migrations/" (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
template_id UUID REFERENCES templates(id) ON DELETE CASCADE,
repository_url VARCHAR(500) NOT NULL,
@ -21,13 +21,13 @@ CREATE TABLE IF NOT EXISTS github_repositories (
-- Create indexes for better performance
CREATE INDEX IF NOT EXISTS idx_github_repos_template_id ON github_repositories(template_id);
CREATE INDEX IF NOT EXISTS idx_github_repos_owner_name ON github_repositories(owner_name);
CREATE INDEX IF NOT EXISTS idx_github_repos_template_id ON "github_repositories@migrations/"(template_id);
CREATE INDEX IF NOT EXISTS idx_github_repos_owner_name ON "github_repositories@migrations/"(owner_name);
CREATE INDEX IF NOT EXISTS idx_feature_mappings_feature_id ON feature_codebase_mappings(feature_id);
CREATE INDEX IF NOT EXISTS idx_feature_mappings_repo_id ON feature_codebase_mappings(repository_id);
-- Add trigger to update timestamp
CREATE TRIGGER update_github_repos_updated_at BEFORE UPDATE ON github_repositories
CREATE TRIGGER update_github_repos_updated_at BEFORE UPDATE ON "github_repositories@migrations/"
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
-- =============================================

6
services/git-integration/src/migrations/002_repository_file_storage.sql
View File

@ -4,7 +4,7 @@
-- Create table for repository local storage tracking
CREATE TABLE IF NOT EXISTS repository_storage (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES github_repositories(id) ON DELETE CASCADE,
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
local_path TEXT NOT NULL,
storage_status VARCHAR(50) DEFAULT 'pending', -- pending, downloading, completed, error
total_files_count INTEGER DEFAULT 0,
@ -21,7 +21,7 @@ CREATE TABLE IF NOT EXISTS repository_storage (
-- Create table for directory structure
CREATE TABLE IF NOT EXISTS repository_directories (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES github_repositories(id) ON DELETE CASCADE,
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
storage_id UUID REFERENCES repository_storage(id) ON DELETE CASCADE,
parent_directory_id UUID REFERENCES repository_directories(id) ON DELETE CASCADE,
directory_name VARCHAR(255) NOT NULL,
@ -38,7 +38,7 @@ CREATE TABLE IF NOT EXISTS repository_directories (
-- Create table for individual files
CREATE TABLE IF NOT EXISTS repository_files (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES github_repositories(id) ON DELETE CASCADE,
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
storage_id UUID REFERENCES repository_storage(id) ON DELETE CASCADE,
directory_id UUID REFERENCES repository_directories(id) ON DELETE SET NULL,
filename VARCHAR(255) NOT NULL,

6
services/git-integration/src/migrations/003_add_user_id_to_template_refs.sql
View File

@ -2,12 +2,12 @@
-- This ensures we always track which user owns/initiated records tied to a template
-- Add user_id to github_repositories
ALTER TABLE IF EXISTS github_repositories
ALTER TABLE IF EXISTS "github_repositories@migrations/"
ADD COLUMN IF NOT EXISTS user_id UUID REFERENCES users(id) ON DELETE CASCADE;
-- Indexes for github_repositories
CREATE INDEX IF NOT EXISTS idx_github_repos_user_id ON github_repositories(user_id);
CREATE INDEX IF NOT EXISTS idx_github_repos_template_user ON github_repositories(template_id, user_id);
CREATE INDEX IF NOT EXISTS idx_github_repos_user_id ON "github_repositories@migrations/"(user_id);
CREATE INDEX IF NOT EXISTS idx_github_repos_template_user ON "github_repositories@migrations/"(template_id, user_id);
-- Add user_id to feature_codebase_mappings
ALTER TABLE IF EXISTS feature_codebase_mappings

4
services/git-integration/src/migrations/005_webhook_commits.sql
View File

@ -8,7 +8,7 @@ CREATE TABLE IF NOT EXISTS github_webhooks (
action VARCHAR(100),
owner_name VARCHAR(120),
repository_name VARCHAR(200),
repository_id UUID REFERENCES github_repositories(id) ON DELETE SET NULL,
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE SET NULL,
ref VARCHAR(255),
before_sha VARCHAR(64),
after_sha VARCHAR(64),
@ -26,7 +26,7 @@ CREATE INDEX IF NOT EXISTS idx_github_webhooks_event_type ON github_webhooks(eve
-- Track commit SHA transitions per repository to detect changes over time
CREATE TABLE IF NOT EXISTS repository_commit_events (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES github_repositories(id) ON DELETE CASCADE,
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
ref VARCHAR(255),
before_sha VARCHAR(64),
after_sha VARCHAR(64),

2
services/git-integration/src/migrations/006_commit_changes.sql
View File

@ -3,7 +3,7 @@
-- Per-commit details linked to an attached repository
CREATE TABLE IF NOT EXISTS repository_commit_details (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES github_repositories(id) ON DELETE CASCADE,
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
commit_sha VARCHAR(64) NOT NULL,
author_name VARCHAR(200),
author_email VARCHAR(320),

2
services/git-integration/src/migrations/007_add_last_synced_commit.sql
View File

@ -1,5 +1,5 @@
-- 007_add_last_synced_commit.sql
ALTER TABLE github_repositories
ALTER TABLE "github_repositories@migrations/"
ADD COLUMN IF NOT EXISTS last_synced_commit_sha VARCHAR(64),
ADD COLUMN IF NOT EXISTS last_synced_at TIMESTAMP WITH TIME ZONE;

6
services/git-integration/src/migrations/009_provider_webhook_tables.sql
View File

@ -8,7 +8,7 @@ CREATE TABLE IF NOT EXISTS gitlab_webhooks (
action TEXT,
owner_name TEXT NOT NULL,
repository_name TEXT NOT NULL,
repository_id UUID REFERENCES github_repositories(id),
repository_id UUID REFERENCES "github_repositories@migrations/"(id),
ref TEXT,
before_sha TEXT,
after_sha TEXT,
@ -26,7 +26,7 @@ CREATE TABLE IF NOT EXISTS bitbucket_webhooks (
action TEXT,
owner_name TEXT NOT NULL,
repository_name TEXT NOT NULL,
repository_id UUID REFERENCES github_repositories(id),
repository_id UUID REFERENCES "github_repositories@migrations/"(id),
ref TEXT,
before_sha TEXT,
after_sha TEXT,
@ -44,7 +44,7 @@ CREATE TABLE IF NOT EXISTS gitea_webhooks (
action TEXT,
owner_name TEXT NOT NULL,
repository_name TEXT NOT NULL,
repository_id UUID REFERENCES github_repositories(id),
repository_id UUID REFERENCES "github_repositories@migrations/"(id),
ref TEXT,
before_sha TEXT,
after_sha TEXT,

2
services/git-integration/src/migrations/010_remove_template_id.sql
View File

@ -7,7 +7,7 @@ DROP INDEX IF EXISTS idx_github_repos_template_user;
DROP INDEX IF EXISTS idx_feature_mappings_template_user;
-- Remove template_id column from github_repositories table
ALTER TABLE IF EXISTS github_repositories
ALTER TABLE IF EXISTS "github_repositories@migrations/"
DROP COLUMN IF EXISTS template_id;
-- Remove template_id column from feature_codebase_mappings table

4
services/git-integration/src/migrations/012_add_user_id_to_github_repositories.sql
View File

@ -1,10 +1,10 @@
-- Migration 012: Track which user attached/downloaded a repository
-- Add user_id to github_repositories to associate records with the initiating user
ALTER TABLE github_repositories
ALTER TABLE "github_repositories@migrations/"
ADD COLUMN IF NOT EXISTS user_id UUID REFERENCES users(id) ON DELETE SET NULL;
-- Helpful index for filtering user-owned repositories
CREATE INDEX IF NOT EXISTS idx_github_repositories_user_id ON github_repositories(user_id);
CREATE INDEX IF NOT EXISTS idx_github_repositories_user_id ON "github_repositories@migrations/"(user_id);

30
services/git-integration/src/migrations/013_repository_commit_details.sql Normal file
View File

@ -0,0 +1,30 @@
-- Migration 013: Repository Commit Details and Files
-- This migration adds commit tracking tables that are missing from the current schema
-- Per-commit details linked to an attached repository
CREATE TABLE IF NOT EXISTS repository_commit_details (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
commit_sha VARCHAR(64) NOT NULL,
author_name VARCHAR(200),
author_email VARCHAR(320),
message TEXT,
url TEXT,
committed_at TIMESTAMP DEFAULT NOW(),
created_at TIMESTAMP DEFAULT NOW()
);
-- Per-file changes for each commit
CREATE TABLE IF NOT EXISTS repository_commit_files (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
commit_id UUID REFERENCES repository_commit_details(id) ON DELETE CASCADE,
change_type VARCHAR(20) NOT NULL, -- added | modified | removed
file_path TEXT NOT NULL,
created_at TIMESTAMP DEFAULT NOW()
);
-- Create indexes for performance
CREATE UNIQUE INDEX IF NOT EXISTS uq_repo_commit_sha ON repository_commit_details(repository_id, commit_sha);
CREATE INDEX IF NOT EXISTS idx_repo_commit_created_at ON repository_commit_details(created_at);
CREATE INDEX IF NOT EXISTS idx_commit_files_commit_id ON repository_commit_files(commit_id);
CREATE INDEX IF NOT EXISTS idx_commit_files_path ON repository_commit_files(file_path);

48
services/git-integration/src/migrations/014_additional_oauth_providers.sql Normal file
View File

@ -0,0 +1,48 @@
-- Migration 014: Additional OAuth Provider Tables
-- This migration adds OAuth token tables for GitLab, Bitbucket, and Gitea
-- GitLab OAuth tokens
CREATE TABLE IF NOT EXISTS gitlab_user_tokens (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
access_token TEXT NOT NULL,
gitlab_username TEXT,
gitlab_user_id TEXT,
scopes JSONB,
expires_at TIMESTAMP NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Bitbucket OAuth tokens
CREATE TABLE IF NOT EXISTS bitbucket_user_tokens (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
access_token TEXT NOT NULL,
bitbucket_username TEXT,
bitbucket_user_id TEXT,
scopes JSONB,
expires_at TIMESTAMP NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Gitea OAuth tokens
CREATE TABLE IF NOT EXISTS gitea_user_tokens (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
access_token TEXT NOT NULL,
gitea_username TEXT,
gitea_user_id TEXT,
scopes JSONB,
expires_at TIMESTAMP NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Add triggers for updated_at columns
CREATE TRIGGER update_gitlab_user_tokens_updated_at BEFORE UPDATE ON gitlab_user_tokens
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
CREATE TRIGGER update_bitbucket_user_tokens_updated_at BEFORE UPDATE ON bitbucket_user_tokens
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
CREATE TRIGGER update_gitea_user_tokens_updated_at BEFORE UPDATE ON gitea_user_tokens
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();

149
services/git-integration/src/migrations/015_diff_storage_system.sql Normal file
View File

@ -0,0 +1,149 @@
-- Migration 015: Git Diff Storage System
-- This migration adds tables for storing git diffs with size-based storage strategy
-- Store actual diff content with size-based strategy
CREATE TABLE IF NOT EXISTS diff_contents (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
commit_id UUID REFERENCES repository_commit_details(id) ON DELETE CASCADE,
file_change_id UUID REFERENCES repository_commit_files(id) ON DELETE CASCADE,
-- Diff metadata
diff_header TEXT, -- "diff --git a/file.py b/file.py"
diff_size_bytes INTEGER NOT NULL,
-- Storage strategy based on size
storage_type VARCHAR(20) NOT NULL, -- 'external' (all diffs stored on disk)
-- Store reference only (path on disk)
external_storage_path TEXT NOT NULL,
external_storage_provider VARCHAR(50) DEFAULT 'local', -- 'local', 's3', 'gcs'
-- File information
file_path TEXT NOT NULL,
change_type VARCHAR(20) NOT NULL, -- 'added', 'modified', 'deleted', 'renamed'
-- Processing status
processing_status VARCHAR(20) DEFAULT 'pending', -- 'pending', 'processed', 'failed'
processing_error TEXT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- Diff Processing Queue (for background processing)
CREATE TABLE IF NOT EXISTS diff_processing_queue (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
commit_id UUID REFERENCES repository_commit_details(id) ON DELETE CASCADE,
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
-- Processing metadata
queue_status VARCHAR(20) DEFAULT 'pending', -- 'pending', 'processing', 'completed', 'failed'
priority INTEGER DEFAULT 0, -- Higher number = higher priority
retry_count INTEGER DEFAULT 0,
max_retries INTEGER DEFAULT 3,
-- Git diff parameters
from_sha VARCHAR(64),
to_sha VARCHAR(64),
repo_local_path TEXT NOT NULL,
-- Processing results
processed_at TIMESTAMP,
error_message TEXT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- Diff Statistics (for monitoring and optimization)
CREATE TABLE IF NOT EXISTS diff_statistics (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
-- Statistics period
period_start TIMESTAMP NOT NULL,
period_end TIMESTAMP NOT NULL,
-- Count statistics
total_commits INTEGER DEFAULT 0,
total_files_changed INTEGER DEFAULT 0,
total_diffs_processed INTEGER DEFAULT 0,
-- Size statistics
total_diff_size_bytes BIGINT DEFAULT 0,
avg_diff_size_bytes DECIMAL(10,2) DEFAULT 0,
max_diff_size_bytes BIGINT DEFAULT 0,
-- Storage type breakdown
diffs_stored_external INTEGER DEFAULT 0,
-- Performance metrics
avg_processing_time_ms DECIMAL(10,2) DEFAULT 0,
failed_processing_count INTEGER DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW()
);
-- Indexes for Performance
CREATE INDEX IF NOT EXISTS idx_diff_contents_commit_id ON diff_contents(commit_id);
CREATE INDEX IF NOT EXISTS idx_diff_contents_file_change_id ON diff_contents(file_change_id);
CREATE INDEX IF NOT EXISTS idx_diff_contents_storage_type ON diff_contents(storage_type);
CREATE INDEX IF NOT EXISTS idx_diff_contents_file_path ON diff_contents(file_path);
CREATE INDEX IF NOT EXISTS idx_diff_contents_change_type ON diff_contents(change_type);
CREATE INDEX IF NOT EXISTS idx_diff_contents_processing_status ON diff_contents(processing_status);
CREATE INDEX IF NOT EXISTS idx_diff_contents_created_at ON diff_contents(created_at);
CREATE INDEX IF NOT EXISTS idx_diff_queue_status ON diff_processing_queue(queue_status);
CREATE INDEX IF NOT EXISTS idx_diff_queue_priority ON diff_processing_queue(priority DESC);
CREATE INDEX IF NOT EXISTS idx_diff_queue_repository_id ON diff_processing_queue(repository_id);
CREATE INDEX IF NOT EXISTS idx_diff_queue_created_at ON diff_processing_queue(created_at);
CREATE INDEX IF NOT EXISTS idx_diff_stats_repository_id ON diff_statistics(repository_id);
CREATE INDEX IF NOT EXISTS idx_diff_stats_period ON diff_statistics(period_start, period_end);
-- Triggers for Updated At Columns
CREATE TRIGGER update_diff_contents_updated_at BEFORE UPDATE ON diff_contents
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
CREATE TRIGGER update_diff_queue_updated_at BEFORE UPDATE ON diff_processing_queue
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
-- Helper Functions
-- Function to get diff storage statistics for a repository
CREATE OR REPLACE FUNCTION get_repository_diff_stats(repo_id UUID, days_back INTEGER DEFAULT 30)
RETURNS TABLE (
total_diffs BIGINT,
total_size_bytes BIGINT,
avg_size_bytes DECIMAL(10,2),
storage_breakdown JSONB
) AS $$
BEGIN
RETURN QUERY
SELECT
COUNT(*)::BIGINT as total_diffs,
COALESCE(SUM(dc.diff_size_bytes), 0)::BIGINT as total_size_bytes,
COALESCE(AVG(dc.diff_size_bytes), 0)::DECIMAL(10,2) as avg_size_bytes,
jsonb_build_object(
'external', COUNT(*) FILTER (WHERE dc.storage_type = 'external')
) as storage_breakdown
FROM diff_contents dc
JOIN repository_commit_details rcd ON dc.commit_id = rcd.id
WHERE rcd.repository_id = repo_id
AND dc.created_at >= NOW() - INTERVAL '1 day' * days_back;
END;
$$ LANGUAGE plpgsql;
-- Function to clean up old diff processing queue entries
CREATE OR REPLACE FUNCTION cleanup_old_diff_queue_entries(days_back INTEGER DEFAULT 7)
RETURNS INTEGER AS $$
DECLARE
deleted_count INTEGER;
BEGIN
DELETE FROM diff_processing_queue
WHERE created_at < NOW() - INTERVAL '1 day' * days_back
AND queue_status IN ('completed', 'failed');
GET DIAGNOSTICS deleted_count = ROW_COUNT;
RETURN deleted_count;
END;
$$ LANGUAGE plpgsql;

25
services/git-integration/src/migrations/016_missing_columns_and_indexes.sql Normal file
View File

@ -0,0 +1,25 @@
-- Migration 016: Missing Columns and Additional Indexes
-- This migration adds missing columns and indexes from the provided migrations
-- Add missing column to github_repositories if it doesn't exist
ALTER TABLE "github_repositories@migrations/"
ADD COLUMN IF NOT EXISTS last_synced_commit_sha VARCHAR(64);
-- Add missing ID column to repository_files if it doesn't exist
ALTER TABLE repository_files
ADD COLUMN IF NOT EXISTS id UUID PRIMARY KEY DEFAULT uuid_generate_v4();
-- Additional indexes for better performance that were missing
CREATE INDEX IF NOT EXISTS idx_repo_directories_level ON repository_directories(level);
CREATE INDEX IF NOT EXISTS idx_repo_directories_relative_path ON repository_directories(relative_path);
CREATE INDEX IF NOT EXISTS idx_repo_files_extension ON repository_files(file_extension);
CREATE INDEX IF NOT EXISTS idx_repo_files_filename ON repository_files(filename);
CREATE INDEX IF NOT EXISTS idx_repo_files_relative_path ON repository_files(relative_path);
CREATE INDEX IF NOT EXISTS idx_repo_files_is_binary ON repository_files(is_binary);
-- Webhook indexes that might be missing
CREATE INDEX IF NOT EXISTS idx_bitbucket_webhooks_event_type ON bitbucket_webhooks(event_type);
CREATE INDEX IF NOT EXISTS idx_gitea_webhooks_repository_id ON gitea_webhooks(repository_id);
CREATE INDEX IF NOT EXISTS idx_gitea_webhooks_created_at ON gitea_webhooks(created_at);
CREATE INDEX IF NOT EXISTS idx_gitea_webhooks_event_type ON gitea_webhooks(event_type);

569
services/git-integration/src/migrations/017_complete_schema_from_provided_migrations.sql Normal file
View File

@ -0,0 +1,569 @@
-- Migration 017: Complete Schema from Provided Migrations
-- This migration creates all tables from the provided 001_github_integration.sql and 002_diff_storage.sql files
-- =============================================
-- Core Repository Tables
-- =============================================
-- Create table for GitHub repositories (enhanced version from provided migration)
CREATE TABLE IF NOT EXISTS "github_repositories@migrations/" (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
template_id UUID, -- References templates(id) but table may not exist
repository_url VARCHAR(500) NOT NULL,
repository_name VARCHAR(200) NOT NULL,
owner_name VARCHAR(100) NOT NULL,
branch_name VARCHAR(100) DEFAULT 'main',
is_public BOOLEAN DEFAULT true,
requires_auth BOOLEAN DEFAULT false,
last_synced_at TIMESTAMP,
sync_status VARCHAR(50) DEFAULT 'pending',
metadata JSONB,
codebase_analysis JSONB,
last_synced_commit_sha VARCHAR(64),
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- =============================================
-- Repository File Storage Tables
-- =============================================
-- Create table for repository local storage tracking
CREATE TABLE IF NOT EXISTS repository_storage (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
local_path TEXT NOT NULL,
storage_status VARCHAR(50) DEFAULT 'pending', -- pending, downloading, completed, error
total_files_count INTEGER DEFAULT 0,
total_directories_count INTEGER DEFAULT 0,
total_size_bytes BIGINT DEFAULT 0,
download_started_at TIMESTAMP,
download_completed_at TIMESTAMP,
last_verified_at TIMESTAMP,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
UNIQUE(repository_id)
);
-- Create table for directory structure
CREATE TABLE IF NOT EXISTS repository_directories (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
storage_id UUID REFERENCES repository_storage(id) ON DELETE CASCADE,
parent_directory_id UUID REFERENCES repository_directories(id) ON DELETE CASCADE,
directory_name VARCHAR(255) NOT NULL,
relative_path TEXT NOT NULL, -- path from repository root
absolute_path TEXT NOT NULL, -- full local filesystem path
level INTEGER DEFAULT 0, -- depth in hierarchy (0 = root)
files_count INTEGER DEFAULT 0,
subdirectories_count INTEGER DEFAULT 0,
total_size_bytes BIGINT DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- Create table for individual files
CREATE TABLE IF NOT EXISTS repository_files (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
storage_id UUID REFERENCES repository_storage(id) ON DELETE CASCADE,
directory_id UUID REFERENCES repository_directories(id) ON DELETE SET NULL,
filename VARCHAR(255) NOT NULL,
file_extension VARCHAR(50),
relative_path TEXT NOT NULL, -- path from repository root
absolute_path TEXT NOT NULL, -- full local filesystem path
file_size_bytes BIGINT DEFAULT 0,
file_hash VARCHAR(64), -- SHA-256 hash for integrity
mime_type VARCHAR(100),
is_binary BOOLEAN DEFAULT false,
encoding VARCHAR(50) DEFAULT 'utf-8',
github_sha VARCHAR(40), -- GitHub blob SHA for tracking changes
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- =============================================
-- Webhook Tables
-- =============================================
-- GitHub webhooks table
CREATE TABLE IF NOT EXISTS github_webhooks (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
delivery_id VARCHAR(120),
event_type VARCHAR(100) NOT NULL,
action VARCHAR(100),
owner_name VARCHAR(120),
repository_name VARCHAR(200),
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE SET NULL,
ref VARCHAR(255),
before_sha VARCHAR(64),
after_sha VARCHAR(64),
commit_count INTEGER,
payload JSONB NOT NULL,
processed_at TIMESTAMP,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- GitLab webhooks table
CREATE TABLE IF NOT EXISTS gitlab_webhooks (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
delivery_id TEXT,
event_type TEXT NOT NULL,
action TEXT,
owner_name TEXT NOT NULL,
repository_name TEXT NOT NULL,
repository_id UUID REFERENCES "github_repositories@migrations/"(id),
ref TEXT,
before_sha TEXT,
after_sha TEXT,
commit_count INTEGER DEFAULT 0,
payload JSONB,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Bitbucket webhooks table
CREATE TABLE IF NOT EXISTS bitbucket_webhooks (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
delivery_id TEXT,
event_type TEXT NOT NULL,
action TEXT,
owner_name TEXT NOT NULL,
repository_name TEXT NOT NULL,
repository_id UUID REFERENCES "github_repositories@migrations/"(id),
ref TEXT,
before_sha TEXT,
after_sha TEXT,
commit_count INTEGER DEFAULT 0,
payload JSONB,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Gitea webhooks table
CREATE TABLE IF NOT EXISTS gitea_webhooks (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
delivery_id TEXT,
event_type TEXT NOT NULL,
action TEXT,
owner_name TEXT NOT NULL,
repository_name TEXT NOT NULL,
repository_id UUID REFERENCES "github_repositories@migrations/"(id),
ref TEXT,
before_sha TEXT,
after_sha TEXT,
commit_count INTEGER DEFAULT 0,
payload JSONB,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- =============================================
-- Commit Tracking Tables
-- =============================================
-- Per-commit details linked to an attached repository
CREATE TABLE IF NOT EXISTS repository_commit_details (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
commit_sha VARCHAR(64) NOT NULL,
author_name VARCHAR(200),
author_email VARCHAR(320),
message TEXT,
url TEXT,
committed_at TIMESTAMP DEFAULT NOW(),
created_at TIMESTAMP DEFAULT NOW()
);
-- Per-file changes for each commit
CREATE TABLE IF NOT EXISTS repository_commit_files (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
commit_id UUID REFERENCES repository_commit_details(id) ON DELETE CASCADE,
change_type VARCHAR(20) NOT NULL, -- added | modified | removed
file_path TEXT NOT NULL,
created_at TIMESTAMP DEFAULT NOW()
);
-- =============================================
-- OAuth Token Tables
-- =============================================
-- GitHub OAuth tokens
CREATE TABLE IF NOT EXISTS github_user_tokens (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
access_token TEXT NOT NULL,
github_username VARCHAR(100) NOT NULL,
github_user_id INTEGER NOT NULL,
scopes JSONB,
expires_at TIMESTAMP,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- GitLab OAuth tokens
CREATE TABLE IF NOT EXISTS gitlab_user_tokens (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
access_token TEXT NOT NULL,
gitlab_username TEXT,
gitlab_user_id TEXT,
scopes JSONB,
expires_at TIMESTAMP NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Bitbucket OAuth tokens
CREATE TABLE IF NOT EXISTS bitbucket_user_tokens (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
access_token TEXT NOT NULL,
bitbucket_username TEXT,
bitbucket_user_id TEXT,
scopes JSONB,
expires_at TIMESTAMP NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Gitea OAuth tokens
CREATE TABLE IF NOT EXISTS gitea_user_tokens (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
access_token TEXT NOT NULL,
gitea_username TEXT,
gitea_user_id TEXT,
scopes JSONB,
expires_at TIMESTAMP NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- =============================================
-- Diff Storage Tables (from 002_diff_storage.sql)
-- =============================================
-- Store actual diff content with size-based strategy
CREATE TABLE IF NOT EXISTS diff_contents (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
commit_id UUID REFERENCES repository_commit_details(id) ON DELETE CASCADE,
file_change_id UUID REFERENCES repository_commit_files(id) ON DELETE CASCADE,
-- Diff metadata
diff_header TEXT, -- "diff --git a/file.py b/file.py"
diff_size_bytes INTEGER NOT NULL,
-- Storage strategy based on size
storage_type VARCHAR(20) NOT NULL, -- 'external' (all diffs stored on disk)
-- Store reference only (path on disk)
external_storage_path TEXT NOT NULL,
external_storage_provider VARCHAR(50) DEFAULT 'local', -- 'local', 's3', 'gcs'
-- File information
file_path TEXT NOT NULL,
change_type VARCHAR(20) NOT NULL, -- 'added', 'modified', 'deleted', 'renamed'
-- Processing status
processing_status VARCHAR(20) DEFAULT 'pending', -- 'pending', 'processed', 'failed'
processing_error TEXT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- Diff Processing Queue (for background processing)
CREATE TABLE IF NOT EXISTS diff_processing_queue (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
commit_id UUID REFERENCES repository_commit_details(id) ON DELETE CASCADE,
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
-- Processing metadata
queue_status VARCHAR(20) DEFAULT 'pending', -- 'pending', 'processing', 'completed', 'failed'
priority INTEGER DEFAULT 0, -- Higher number = higher priority
retry_count INTEGER DEFAULT 0,
max_retries INTEGER DEFAULT 3,
-- Git diff parameters
from_sha VARCHAR(64),
to_sha VARCHAR(64),
repo_local_path TEXT NOT NULL,
-- Processing results
processed_at TIMESTAMP,
error_message TEXT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- Diff Statistics (for monitoring and optimization)
CREATE TABLE IF NOT EXISTS diff_statistics (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
repository_id UUID REFERENCES "github_repositories@migrations/"(id) ON DELETE CASCADE,
-- Statistics period
period_start TIMESTAMP NOT NULL,
period_end TIMESTAMP NOT NULL,
-- Count statistics
total_commits INTEGER DEFAULT 0,
total_files_changed INTEGER DEFAULT 0,
total_diffs_processed INTEGER DEFAULT 0,
-- Size statistics
total_diff_size_bytes BIGINT DEFAULT 0,
avg_diff_size_bytes DECIMAL(10,2) DEFAULT 0,
max_diff_size_bytes BIGINT DEFAULT 0,
-- Storage type breakdown
diffs_stored_external INTEGER DEFAULT 0,
-- Performance metrics
avg_processing_time_ms DECIMAL(10,2) DEFAULT 0,
failed_processing_count INTEGER DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW()
);
-- =============================================
-- Indexes for Performance
-- =============================================
-- GitHub repositories indexes
CREATE INDEX IF NOT EXISTS idx_github_repos_template_id ON "github_repositories@migrations/"(template_id);
CREATE INDEX IF NOT EXISTS idx_github_repos_owner_name ON "github_repositories@migrations/"(owner_name);
-- Repository storage indexes
CREATE INDEX IF NOT EXISTS idx_repository_storage_repo_id ON repository_storage(repository_id);
CREATE INDEX IF NOT EXISTS idx_repository_storage_status ON repository_storage(storage_status);
-- Repository directories indexes
CREATE INDEX IF NOT EXISTS idx_repo_directories_repo_id ON repository_directories(repository_id);
CREATE INDEX IF NOT EXISTS idx_repo_directories_parent_id ON repository_directories(parent_directory_id);
CREATE INDEX IF NOT EXISTS idx_repo_directories_storage_id ON repository_directories(storage_id);
CREATE INDEX IF NOT EXISTS idx_repo_directories_level ON repository_directories(level);
CREATE INDEX IF NOT EXISTS idx_repo_directories_relative_path ON repository_directories(relative_path);
-- Repository files indexes
CREATE INDEX IF NOT EXISTS idx_repo_files_repo_id ON repository_files(repository_id);
CREATE INDEX IF NOT EXISTS idx_repo_files_directory_id ON repository_files(directory_id);
CREATE INDEX IF NOT EXISTS idx_repo_files_storage_id ON repository_files(storage_id);
CREATE INDEX IF NOT EXISTS idx_repo_files_extension ON repository_files(file_extension);
CREATE INDEX IF NOT EXISTS idx_repo_files_filename ON repository_files(filename);
CREATE INDEX IF NOT EXISTS idx_repo_files_relative_path ON repository_files(relative_path);
CREATE INDEX IF NOT EXISTS idx_repo_files_is_binary ON repository_files(is_binary);
-- GitHub webhooks indexes
CREATE INDEX IF NOT EXISTS idx_github_webhooks_delivery_id ON github_webhooks(delivery_id);
CREATE INDEX IF NOT EXISTS idx_github_webhooks_repo ON github_webhooks(owner_name, repository_name);
CREATE INDEX IF NOT EXISTS idx_github_webhooks_event_type ON github_webhooks(event_type);
-- GitLab webhooks indexes
CREATE INDEX IF NOT EXISTS idx_gitlab_webhooks_repository_id ON gitlab_webhooks(repository_id);
CREATE INDEX IF NOT EXISTS idx_gitlab_webhooks_created_at ON gitlab_webhooks(created_at);
CREATE INDEX IF NOT EXISTS idx_gitlab_webhooks_event_type ON gitlab_webhooks(event_type);
-- Bitbucket webhooks indexes
CREATE INDEX IF NOT EXISTS idx_bitbucket_webhooks_repository_id ON bitbucket_webhooks(repository_id);
CREATE INDEX IF NOT EXISTS idx_bitbucket_webhooks_created_at ON bitbucket_webhooks(created_at);
CREATE INDEX IF NOT EXISTS idx_bitbucket_webhooks_event_type ON bitbucket_webhooks(event_type);
-- Gitea webhooks indexes
CREATE INDEX IF NOT EXISTS idx_gitea_webhooks_repository_id ON gitea_webhooks(repository_id);
CREATE INDEX IF NOT EXISTS idx_gitea_webhooks_created_at ON gitea_webhooks(created_at);
CREATE INDEX IF NOT EXISTS idx_gitea_webhooks_event_type ON gitea_webhooks(event_type);
-- Commit details indexes
CREATE UNIQUE INDEX IF NOT EXISTS uq_repo_commit_sha ON repository_commit_details(repository_id, commit_sha);
CREATE INDEX IF NOT EXISTS idx_repo_commit_created_at ON repository_commit_details(created_at);
-- Commit files indexes
CREATE INDEX IF NOT EXISTS idx_commit_files_commit_id ON repository_commit_files(commit_id);
CREATE INDEX IF NOT EXISTS idx_commit_files_path ON repository_commit_files(file_path);
-- OAuth token indexes
CREATE INDEX IF NOT EXISTS idx_github_user_tokens_github_username ON github_user_tokens(github_username);
-- Diff contents indexes
CREATE INDEX IF NOT EXISTS idx_diff_contents_commit_id ON diff_contents(commit_id);
CREATE INDEX IF NOT EXISTS idx_diff_contents_file_change_id ON diff_contents(file_change_id);
CREATE INDEX IF NOT EXISTS idx_diff_contents_storage_type ON diff_contents(storage_type);
CREATE INDEX IF NOT EXISTS idx_diff_contents_file_path ON diff_contents(file_path);
CREATE INDEX IF NOT EXISTS idx_diff_contents_change_type ON diff_contents(change_type);
CREATE INDEX IF NOT EXISTS idx_diff_contents_processing_status ON diff_contents(processing_status);
CREATE INDEX IF NOT EXISTS idx_diff_contents_created_at ON diff_contents(created_at);
-- Processing queue indexes
CREATE INDEX IF NOT EXISTS idx_diff_queue_status ON diff_processing_queue(queue_status);
CREATE INDEX IF NOT EXISTS idx_diff_queue_priority ON diff_processing_queue(priority DESC);
CREATE INDEX IF NOT EXISTS idx_diff_queue_repository_id ON diff_processing_queue(repository_id);
CREATE INDEX IF NOT EXISTS idx_diff_queue_created_at ON diff_processing_queue(created_at);
-- Statistics indexes
CREATE INDEX IF NOT EXISTS idx_diff_stats_repository_id ON diff_statistics(repository_id);
CREATE INDEX IF NOT EXISTS idx_diff_stats_period ON diff_statistics(period_start, period_end);
-- =============================================
-- Triggers for Updated At Columns
-- =============================================
-- Create update function if it doesn't exist
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;
$$ language 'plpgsql';
-- GitHub repositories trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_github_repos_updated_at'
) THEN
CREATE TRIGGER update_github_repos_updated_at BEFORE UPDATE ON "github_repositories@migrations/"
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- Repository storage trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_repository_storage_updated_at'
) THEN
CREATE TRIGGER update_repository_storage_updated_at BEFORE UPDATE ON repository_storage
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- Repository directories trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_repository_directories_updated_at'
) THEN
CREATE TRIGGER update_repository_directories_updated_at BEFORE UPDATE ON repository_directories
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- Repository files trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_repository_files_updated_at'
) THEN
CREATE TRIGGER update_repository_files_updated_at BEFORE UPDATE ON repository_files
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- GitHub webhooks trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_github_webhooks_updated_at'
) THEN
CREATE TRIGGER update_github_webhooks_updated_at BEFORE UPDATE ON github_webhooks
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- GitLab user tokens trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_gitlab_user_tokens_updated_at'
) THEN
CREATE TRIGGER update_gitlab_user_tokens_updated_at BEFORE UPDATE ON gitlab_user_tokens
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- Bitbucket user tokens trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_bitbucket_user_tokens_updated_at'
) THEN
CREATE TRIGGER update_bitbucket_user_tokens_updated_at BEFORE UPDATE ON bitbucket_user_tokens
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- Gitea user tokens trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_gitea_user_tokens_updated_at'
) THEN
CREATE TRIGGER update_gitea_user_tokens_updated_at BEFORE UPDATE ON gitea_user_tokens
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- Diff contents trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_diff_contents_updated_at'
) THEN
CREATE TRIGGER update_diff_contents_updated_at BEFORE UPDATE ON diff_contents
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- Processing queue trigger
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_trigger WHERE tgname = 'update_diff_queue_updated_at'
) THEN
CREATE TRIGGER update_diff_queue_updated_at BEFORE UPDATE ON diff_processing_queue
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
END IF;
END $$;
-- =============================================
-- Helper Functions
-- =============================================
-- Function to get diff storage statistics for a repository
CREATE OR REPLACE FUNCTION get_repository_diff_stats(repo_id UUID, days_back INTEGER DEFAULT 30)
RETURNS TABLE (
total_diffs BIGINT,
total_size_bytes BIGINT,
avg_size_bytes DECIMAL(10,2),
storage_breakdown JSONB
) AS $$
BEGIN
RETURN QUERY
SELECT
COUNT(*)::BIGINT as total_diffs,
COALESCE(SUM(dc.diff_size_bytes), 0)::BIGINT as total_size_bytes,
COALESCE(AVG(dc.diff_size_bytes), 0)::DECIMAL(10,2) as avg_size_bytes,
jsonb_build_object(
'external', COUNT(*) FILTER (WHERE dc.storage_type = 'external')
) as storage_breakdown
FROM diff_contents dc
JOIN repository_commit_details rcd ON dc.commit_id = rcd.id
WHERE rcd.repository_id = repo_id
AND dc.created_at >= NOW() - INTERVAL '1 day' * days_back;
END;
$$ LANGUAGE plpgsql;
-- Function to clean up old diff processing queue entries
CREATE OR REPLACE FUNCTION cleanup_old_diff_queue_entries(days_back INTEGER DEFAULT 7)
RETURNS INTEGER AS $$
DECLARE
deleted_count INTEGER;
BEGIN
DELETE FROM diff_processing_queue
WHERE created_at < NOW() - INTERVAL '1 day' * days_back
AND queue_status IN ('completed', 'failed');
GET DIAGNOSTICS deleted_count = ROW_COUNT;
RETURN deleted_count;
END;
$$ LANGUAGE plpgsql;

79
services/git-integration/src/models/commit-details.model.js Normal file
View File

@ -0,0 +1,79 @@
const { v4: uuidv4 } = require('uuid');
const database = require('../config/database');
class CommitDetailsModel {
constructor() {
this.db = database;
}
async createCommitDetail(commitData) {
const {
repository_id,
commit_sha,
author_name,
author_email,
message,
url,
committed_at
} = commitData;
const query = `
INSERT INTO repository_commit_details (
id, repository_id, commit_sha, author_name, author_email,
message, url, committed_at, created_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, NOW())
ON CONFLICT (repository_id, commit_sha)
DO UPDATE SET
author_name = EXCLUDED.author_name,
author_email = EXCLUDED.author_email,
message = EXCLUDED.message,
url = EXCLUDED.url,
committed_at = EXCLUDED.committed_at
RETURNING *
`;
const values = [
uuidv4(),
repository_id,
commit_sha,
author_name,
author_email,
message,
url,
committed_at || new Date()
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async getCommitsByRepository(repository_id, limit = 50, offset = 0) {
const query = `
SELECT * FROM repository_commit_details
WHERE repository_id = $1
ORDER BY committed_at DESC
LIMIT $2 OFFSET $3
`;
const result = await this.db.query(query, [repository_id, limit, offset]);
return result.rows;
}
async getCommitBySha(repository_id, commit_sha) {
const query = `
SELECT * FROM repository_commit_details
WHERE repository_id = $1 AND commit_sha = $2
`;
const result = await this.db.query(query, [repository_id, commit_sha]);
return result.rows[0];
}
async deleteCommitsByRepository(repository_id) {
const query = `DELETE FROM repository_commit_details WHERE repository_id = $1`;
const result = await this.db.query(query, [repository_id]);
return result.rowCount;
}
}
module.exports = CommitDetailsModel;

102
services/git-integration/src/models/commit-files.model.js Normal file
View File

@ -0,0 +1,102 @@
const { v4: uuidv4 } = require('uuid');
const database = require('../config/database');
class CommitFilesModel {
constructor() {
this.db = database;
}
async createCommitFile(fileData) {
const {
commit_id,
change_type,
file_path
} = fileData;
const query = `
INSERT INTO repository_commit_files (
id, commit_id, change_type, file_path, created_at
) VALUES ($1, $2, $3, $4, NOW())
RETURNING *
`;
const values = [
uuidv4(),
commit_id,
change_type,
file_path
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async createMultipleCommitFiles(commit_id, files) {
if (!files || files.length === 0) return [];
const values = [];
const placeholders = [];
files.forEach((file, index) => {
const baseIndex = index * 4;
placeholders.push(`($${baseIndex + 1}, $${baseIndex + 2}, $${baseIndex + 3}, $${baseIndex + 4}, NOW())`);
values.push(uuidv4(), commit_id, file.change_type, file.file_path);
});
const query = `
INSERT INTO repository_commit_files (
id, commit_id, change_type, file_path, created_at
) VALUES ${placeholders.join(', ')}
RETURNING *
`;
const result = await this.db.query(query, values);
return result.rows;
}
async getFilesByCommit(commit_id) {
const query = `
SELECT * FROM repository_commit_files
WHERE commit_id = $1
ORDER BY file_path
`;
const result = await this.db.query(query, [commit_id]);
return result.rows;
}
async getFilesByRepository(repository_id, limit = 100, offset = 0) {
const query = `
SELECT rcf.*, rcd.commit_sha, rcd.committed_at
FROM repository_commit_files rcf
JOIN repository_commit_details rcd ON rcf.commit_id = rcd.id
WHERE rcd.repository_id = $1
ORDER BY rcd.committed_at DESC, rcf.file_path
LIMIT $2 OFFSET $3
`;
const result = await this.db.query(query, [repository_id, limit, offset]);
return result.rows;
}
async getFileChangesByPath(repository_id, file_path) {
const query = `
SELECT rcf.*, rcd.commit_sha, rcd.committed_at, rcd.author_name
FROM repository_commit_files rcf
JOIN repository_commit_details rcd ON rcf.commit_id = rcd.id
WHERE rcd.repository_id = $1 AND rcf.file_path = $2
ORDER BY rcd.committed_at DESC
`;
const result = await this.db.query(query, [repository_id, file_path]);
return result.rows;
}
async deleteFilesByCommit(commit_id) {
const query = `DELETE FROM repository_commit_files WHERE commit_id = $1`;
const result = await this.db.query(query, [commit_id]);
return result.rowCount;
}
}
module.exports = CommitFilesModel;

226
services/git-integration/src/models/diff-storage.model.js Normal file
View File

@ -0,0 +1,226 @@
const { v4: uuidv4 } = require('uuid');
const database = require('../config/database');
class DiffStorageModel {
constructor() {
this.db = database;
}
async createDiffContent(diffData) {
const {
commit_id,
file_change_id,
diff_header,
diff_size_bytes,
storage_type = 'external',
external_storage_path,
external_storage_provider = 'local',
file_path,
change_type,
processing_status = 'pending'
} = diffData;
const query = `
INSERT INTO diff_contents (
id, commit_id, file_change_id, diff_header, diff_size_bytes,
storage_type, external_storage_path, external_storage_provider,
file_path, change_type, processing_status, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, NOW(), NOW())
RETURNING *
`;
const values = [
uuidv4(),
commit_id,
file_change_id,
diff_header,
diff_size_bytes,
storage_type,
external_storage_path,
external_storage_provider,
file_path,
change_type,
processing_status
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async updateProcessingStatus(diff_id, status, error_message = null) {
const query = `
UPDATE diff_contents
SET processing_status = $2, processing_error = $3, updated_at = NOW()
WHERE id = $1
RETURNING *
`;
const result = await this.db.query(query, [diff_id, status, error_message]);
return result.rows[0];
}
async getDiffsByCommit(commit_id) {
const query = `
SELECT * FROM diff_contents
WHERE commit_id = $1
ORDER BY file_path
`;
const result = await this.db.query(query, [commit_id]);
return result.rows;
}
async getDiffsByRepository(repository_id, limit = 50, offset = 0) {
const query = `
SELECT dc.*, rcd.commit_sha, rcd.committed_at
FROM diff_contents dc
JOIN repository_commit_details rcd ON dc.commit_id = rcd.id
WHERE rcd.repository_id = $1
ORDER BY rcd.committed_at DESC, dc.file_path
LIMIT $2 OFFSET $3
`;
const result = await this.db.query(query, [repository_id, limit, offset]);
return result.rows;
}
async getDiffById(diff_id) {
const query = `SELECT * FROM diff_contents WHERE id = $1`;
const result = await this.db.query(query, [diff_id]);
return result.rows[0];
}
// Diff Processing Queue methods
async addToProcessingQueue(queueData) {
const {
commit_id,
repository_id,
queue_status = 'pending',
priority = 0,
from_sha,
to_sha,
repo_local_path
} = queueData;
const query = `
INSERT INTO diff_processing_queue (
id, commit_id, repository_id, queue_status, priority,
from_sha, to_sha, repo_local_path, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, NOW(), NOW())
RETURNING *
`;
const values = [
uuidv4(),
commit_id,
repository_id,
queue_status,
priority,
from_sha,
to_sha,
repo_local_path
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async getNextQueueItem() {
const query = `
SELECT * FROM diff_processing_queue
WHERE queue_status = 'pending'
ORDER BY priority DESC, created_at ASC
LIMIT 1
`;
const result = await this.db.query(query);
return result.rows[0];
}
async updateQueueStatus(queue_id, status, error_message = null) {
const query = `
UPDATE diff_processing_queue
SET queue_status = $2, error_message = $3,
processed_at = CASE WHEN $2 IN ('completed', 'failed') THEN NOW() ELSE processed_at END,
updated_at = NOW()
WHERE id = $1
RETURNING *
`;
const result = await this.db.query(query, [queue_id, status, error_message]);
return result.rows[0];
}
async incrementRetryCount(queue_id) {
const query = `
UPDATE diff_processing_queue
SET retry_count = retry_count + 1, updated_at = NOW()
WHERE id = $1
RETURNING *
`;
const result = await this.db.query(query, [queue_id]);
return result.rows[0];
}
// Statistics methods
async createDiffStatistics(statsData) {
const {
repository_id,
period_start,
period_end,
total_commits = 0,
total_files_changed = 0,
total_diffs_processed = 0,
total_diff_size_bytes = 0,
avg_diff_size_bytes = 0,
max_diff_size_bytes = 0,
diffs_stored_external = 0,
avg_processing_time_ms = 0,
failed_processing_count = 0
} = statsData;
const query = `
INSERT INTO diff_statistics (
id, repository_id, period_start, period_end, total_commits,
total_files_changed, total_diffs_processed, total_diff_size_bytes,
avg_diff_size_bytes, max_diff_size_bytes, diffs_stored_external,
avg_processing_time_ms, failed_processing_count, created_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, NOW())
RETURNING *
`;
const values = [
uuidv4(),
repository_id,
period_start,
period_end,
total_commits,
total_files_changed,
total_diffs_processed,
total_diff_size_bytes,
avg_diff_size_bytes,
max_diff_size_bytes,
diffs_stored_external,
avg_processing_time_ms,
failed_processing_count
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async getRepositoryDiffStats(repository_id, days_back = 30) {
const query = `SELECT * FROM get_repository_diff_stats($1, $2)`;
const result = await this.db.query(query, [repository_id, days_back]);
return result.rows[0];
}
async cleanupOldQueueEntries(days_back = 7) {
const query = `SELECT cleanup_old_diff_queue_entries($1)`;
const result = await this.db.query(query, [days_back]);
return result.rows[0].cleanup_old_diff_queue_entries;
}
}
module.exports = DiffStorageModel;

227
services/git-integration/src/models/oauth-tokens.model.js Normal file
View File

@ -0,0 +1,227 @@
const { v4: uuidv4 } = require('uuid');
const database = require('../config/database');
class OAuthTokensModel {
constructor() {
this.db = database;
}
// GitLab OAuth tokens
async createGitLabToken(tokenData) {
const {
access_token,
gitlab_username,
gitlab_user_id,
scopes,
expires_at
} = tokenData;
const query = `
INSERT INTO gitlab_user_tokens (
id, access_token, gitlab_username, gitlab_user_id,
scopes, expires_at, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, NOW(), NOW())
RETURNING *
`;
const values = [
uuidv4(),
access_token,
gitlab_username,
gitlab_user_id,
scopes,
expires_at
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async getGitLabTokenByUsername(gitlab_username) {
const query = `
SELECT * FROM gitlab_user_tokens
WHERE gitlab_username = $1
ORDER BY created_at DESC
LIMIT 1
`;
const result = await this.db.query(query, [gitlab_username]);
return result.rows[0];
}
async updateGitLabToken(token_id, tokenData) {
const {
access_token,
scopes,
expires_at
} = tokenData;
const query = `
UPDATE gitlab_user_tokens
SET access_token = $2, scopes = $3, expires_at = $4, updated_at = NOW()
WHERE id = $1
RETURNING *
`;
const result = await this.db.query(query, [token_id, access_token, scopes, expires_at]);
return result.rows[0];
}
// Bitbucket OAuth tokens
async createBitbucketToken(tokenData) {
const {
access_token,
bitbucket_username,
bitbucket_user_id,
scopes,
expires_at
} = tokenData;
const query = `
INSERT INTO bitbucket_user_tokens (
id, access_token, bitbucket_username, bitbucket_user_id,
scopes, expires_at, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, NOW(), NOW())
RETURNING *
`;
const values = [
uuidv4(),
access_token,
bitbucket_username,
bitbucket_user_id,
scopes,
expires_at
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async getBitbucketTokenByUsername(bitbucket_username) {
const query = `
SELECT * FROM bitbucket_user_tokens
WHERE bitbucket_username = $1
ORDER BY created_at DESC
LIMIT 1
`;
const result = await this.db.query(query, [bitbucket_username]);
return result.rows[0];
}
async updateBitbucketToken(token_id, tokenData) {
const {
access_token,
scopes,
expires_at
} = tokenData;
const query = `
UPDATE bitbucket_user_tokens
SET access_token = $2, scopes = $3, expires_at = $4, updated_at = NOW()
WHERE id = $1
RETURNING *
`;
const result = await this.db.query(query, [token_id, access_token, scopes, expires_at]);
return result.rows[0];
}
// Gitea OAuth tokens
async createGiteaToken(tokenData) {
const {
access_token,
gitea_username,
gitea_user_id,
scopes,
expires_at
} = tokenData;
const query = `
INSERT INTO gitea_user_tokens (
id, access_token, gitea_username, gitea_user_id,
scopes, expires_at, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, NOW(), NOW())
RETURNING *
`;
const values = [
uuidv4(),
access_token,
gitea_username,
gitea_user_id,
scopes,
expires_at
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async getGiteaTokenByUsername(gitea_username) {
const query = `
SELECT * FROM gitea_user_tokens
WHERE gitea_username = $1
ORDER BY created_at DESC
LIMIT 1
`;
const result = await this.db.query(query, [gitea_username]);
return result.rows[0];
}
async updateGiteaToken(token_id, tokenData) {
const {
access_token,
scopes,
expires_at
} = tokenData;
const query = `
UPDATE gitea_user_tokens
SET access_token = $2, scopes = $3, expires_at = $4, updated_at = NOW()
WHERE id = $1
RETURNING *
`;
const result = await this.db.query(query, [token_id, access_token, scopes, expires_at]);
return result.rows[0];
}
// Generic methods for all providers
async deleteExpiredTokens() {
const queries = [
'DELETE FROM gitlab_user_tokens WHERE expires_at < NOW()',
'DELETE FROM bitbucket_user_tokens WHERE expires_at < NOW()',
'DELETE FROM gitea_user_tokens WHERE expires_at < NOW()'
];
let totalDeleted = 0;
for (const query of queries) {
const result = await this.db.query(query);
totalDeleted += result.rowCount;
}
return totalDeleted;
}
async getAllTokensByProvider(provider) {
const tableMap = {
'gitlab': 'gitlab_user_tokens',
'bitbucket': 'bitbucket_user_tokens',
'gitea': 'gitea_user_tokens'
};
const tableName = tableMap[provider];
if (!tableName) {
throw new Error(`Unsupported provider: ${provider}`);
}
const query = `SELECT * FROM ${tableName} ORDER BY created_at DESC`;
const result = await this.db.query(query);
return result.rows;
}
}
module.exports = OAuthTokensModel;

273
services/git-integration/src/routes/commits.routes.js Normal file
View File

@ -0,0 +1,273 @@
const express = require('express');
const router = express.Router();
const CommitTrackingService = require('../services/commit-tracking.service');
const DiffStorageModel = require('../models/diff-storage.model');
const EnhancedDiffProcessingService = require('../services/enhanced-diff-processing.service');
const commitTrackingService = new CommitTrackingService();
const diffStorageModel = new DiffStorageModel();
const diffProcessingService = new EnhancedDiffProcessingService();
/**
* GET /api/commits/repository/:repositoryId
* Get commit history for a repository
*/
router.get('/repository/:repositoryId', async (req, res) => {
try {
const { repositoryId } = req.params;
const {
limit = 50,
offset = 0,
include_files = false
} = req.query;
const options = {
limit: parseInt(limit),
offset: parseInt(offset),
include_files: include_files === 'true'
};
const commits = await commitTrackingService.getCommitHistory(repositoryId, options);
res.json({
success: true,
data: commits,
pagination: {
limit: options.limit,
offset: options.offset,
total: commits.length
}
});
} catch (error) {
console.error('Error getting commit history:', error);
res.status(500).json({
success: false,
error: 'Failed to get commit history',
message: error.message
});
}
});
/**
* GET /api/commits/:commitId/files
* Get file changes for a specific commit
*/
router.get('/:commitId/files', async (req, res) => {
try {
const { commitId } = req.params;
const files = await commitTrackingService.commitFilesModel.getFilesByCommit(commitId);
res.json({
success: true,
data: files
});
} catch (error) {
console.error('Error getting commit files:', error);
res.status(500).json({
success: false,
error: 'Failed to get commit files',
message: error.message
});
}
});
/**
* GET /api/commits/repository/:repositoryId/file-history
* Get file change history for a specific file
*/
router.get('/repository/:repositoryId/file-history', async (req, res) => {
try {
const { repositoryId } = req.params;
const { file_path } = req.query;
if (!file_path) {
return res.status(400).json({
success: false,
error: 'file_path query parameter is required'
});
}
const history = await commitTrackingService.getFileHistory(repositoryId, file_path);
res.json({
success: true,
data: history
});
} catch (error) {
console.error('Error getting file history:', error);
res.status(500).json({
success: false,
error: 'Failed to get file history',
message: error.message
});
}
});
/**
* GET /api/commits/repository/:repositoryId/stats
* Get repository commit statistics
*/
router.get('/repository/:repositoryId/stats', async (req, res) => {
try {
const { repositoryId } = req.params;
const stats = await commitTrackingService.getRepositoryStats(repositoryId);
res.json({
success: true,
data: stats
});
} catch (error) {
console.error('Error getting repository stats:', error);
res.status(500).json({
success: false,
error: 'Failed to get repository stats',
message: error.message
});
}
});
/**
* POST /api/commits/process-webhook
* Process commits from webhook payload
*/
router.post('/process-webhook', async (req, res) => {
try {
const { webhookPayload, repositoryId } = req.body;
if (!webhookPayload || !repositoryId) {
return res.status(400).json({
success: false,
error: 'webhookPayload and repositoryId are required'
});
}
const processedCommits = await commitTrackingService.processWebhookCommits(
webhookPayload,
repositoryId
);
res.json({
success: true,
data: {
processed_commits: processedCommits.length,
commits: processedCommits
}
});
} catch (error) {
console.error('Error processing webhook commits:', error);
res.status(500).json({
success: false,
error: 'Failed to process webhook commits',
message: error.message
});
}
});
/**
* GET /api/commits/:commitId/diffs
* Get diff information for a commit
*/
router.get('/:commitId/diffs', async (req, res) => {
try {
const { commitId } = req.params;
const diffs = await diffStorageModel.getDiffsByCommit(commitId);
res.json({
success: true,
data: diffs
});
} catch (error) {
console.error('Error getting commit diffs:', error);
res.status(500).json({
success: false,
error: 'Failed to get commit diffs',
message: error.message
});
}
});
/**
* GET /api/commits/diffs/:diffId/content
* Get actual diff content
*/
router.get('/diffs/:diffId/content', async (req, res) => {
try {
const { diffId } = req.params;
const diffContent = await diffProcessingService.retrieveDiffContent(diffId);
res.set('Content-Type', 'text/plain');
res.send(diffContent);
} catch (error) {
console.error('Error getting diff content:', error);
res.status(500).json({
success: false,
error: 'Failed to get diff content',
message: error.message
});
}
});
/**
* GET /api/commits/repository/:repositoryId/diff-stats
* Get diff statistics for a repository
*/
router.get('/repository/:repositoryId/diff-stats', async (req, res) => {
try {
const { repositoryId } = req.params;
const { days_back = 30 } = req.query;
const stats = await diffStorageModel.getRepositoryDiffStats(
repositoryId,
parseInt(days_back)
);
res.json({
success: true,
data: stats
});
} catch (error) {
console.error('Error getting diff stats:', error);
res.status(500).json({
success: false,
error: 'Failed to get diff stats',
message: error.message
});
}
});
/**
* POST /api/commits/process-diff-queue
* Manually trigger diff processing
*/
router.post('/process-diff-queue', async (req, res) => {
try {
const result = await diffProcessingService.processNextQueueItem();
if (result) {
res.json({
success: true,
data: result,
message: 'Diff processing completed'
});
} else {
res.json({
success: true,
data: null,
message: 'No items in queue to process'
});
}
} catch (error) {
console.error('Error processing diff queue:', error);
res.status(500).json({
success: false,
error: 'Failed to process diff queue',
message: error.message
});
}
});
module.exports = router;

369
services/git-integration/src/routes/enhanced-webhooks.routes.js Normal file
View File

@ -0,0 +1,369 @@
const express = require('express');
const router = express.Router();
const EnhancedWebhookService = require('../services/enhanced-webhook.service');
const webhookService = new EnhancedWebhookService();
/**
* POST /api/webhooks/github
* Handle GitHub webhooks
*/
router.post('/github', async (req, res) => {
try {
const payload = req.body;
const headers = req.headers;
console.log(`Received GitHub webhook: ${headers['x-github-event']}`);
const result = await webhookService.processGitHubWebhook(payload, headers);
res.json({
success: true,
data: result,
message: 'GitHub webhook processed successfully'
});
} catch (error) {
console.error('Error processing GitHub webhook:', error);
res.status(500).json({
success: false,
error: 'Failed to process GitHub webhook',
message: error.message
});
}
});
/**
* POST /api/webhooks/gitlab
* Handle GitLab webhooks
*/
router.post('/gitlab', async (req, res) => {
try {
const payload = req.body;
const headers = req.headers;
console.log(`Received GitLab webhook: ${headers['x-gitlab-event']}`);
const result = await webhookService.processGitLabWebhook(payload, headers);
res.json({
success: true,
data: result,
message: 'GitLab webhook processed successfully'
});
} catch (error) {
console.error('Error processing GitLab webhook:', error);
res.status(500).json({
success: false,
error: 'Failed to process GitLab webhook',
message: error.message
});
}
});
/**
* POST /api/webhooks/bitbucket
* Handle Bitbucket webhooks
*/
router.post('/bitbucket', async (req, res) => {
try {
const payload = req.body;
const headers = req.headers;
console.log(`Received Bitbucket webhook: ${headers['x-event-key']}`);
const result = await webhookService.processBitbucketWebhook(payload, headers);
res.json({
success: true,
data: result,
message: 'Bitbucket webhook processed successfully'
});
} catch (error) {
console.error('Error processing Bitbucket webhook:', error);
res.status(500).json({
success: false,
error: 'Failed to process Bitbucket webhook',
message: error.message
});
}
});
/**
* POST /api/webhooks/gitea
* Handle Gitea webhooks
*/
router.post('/gitea', async (req, res) => {
try {
const payload = req.body;
const headers = req.headers;
console.log(`Received Gitea webhook: ${headers['x-gitea-event']}`);
const result = await webhookService.processGiteaWebhook(payload, headers);
res.json({
success: true,
data: result,
message: 'Gitea webhook processed successfully'
});
} catch (error) {
console.error('Error processing Gitea webhook:', error);
res.status(500).json({
success: false,
error: 'Failed to process Gitea webhook',
message: error.message
});
}
});
/**
* GET /api/webhooks/github/recent
* Get recent GitHub webhooks
*/
router.get('/github/recent', async (req, res) => {
try {
const { limit = 50, offset = 0 } = req.query;
const query = `
SELECT id, delivery_id, event_type, action, owner_name, repository_name,
ref, commit_count, processed_at, created_at
FROM github_webhooks
ORDER BY created_at DESC
LIMIT $1 OFFSET $2
`;
const result = await webhookService.db.query(query, [parseInt(limit), parseInt(offset)]);
res.json({
success: true,
data: result.rows,
pagination: {
limit: parseInt(limit),
offset: parseInt(offset),
total: result.rows.length
}
});
} catch (error) {
console.error('Error getting recent GitHub webhooks:', error);
res.status(500).json({
success: false,
error: 'Failed to get recent GitHub webhooks',
message: error.message
});
}
});
/**
* GET /api/webhooks/gitlab/recent
* Get recent GitLab webhooks
*/
router.get('/gitlab/recent', async (req, res) => {
try {
const { limit = 50, offset = 0 } = req.query;
const query = `
SELECT id, delivery_id, event_type, action, owner_name, repository_name,
ref, commit_count, created_at
FROM gitlab_webhooks
ORDER BY created_at DESC
LIMIT $1 OFFSET $2
`;
const result = await webhookService.db.query(query, [parseInt(limit), parseInt(offset)]);
res.json({
success: true,
data: result.rows,
pagination: {
limit: parseInt(limit),
offset: parseInt(offset),
total: result.rows.length
}
});
} catch (error) {
console.error('Error getting recent GitLab webhooks:', error);
res.status(500).json({
success: false,
error: 'Failed to get recent GitLab webhooks',
message: error.message
});
}
});
/**
* GET /api/webhooks/bitbucket/recent
* Get recent Bitbucket webhooks
*/
router.get('/bitbucket/recent', async (req, res) => {
try {
const { limit = 50, offset = 0 } = req.query;
const query = `
SELECT id, delivery_id, event_type, action, owner_name, repository_name,
ref, commit_count, created_at
FROM bitbucket_webhooks
ORDER BY created_at DESC
LIMIT $1 OFFSET $2
`;
const result = await webhookService.db.query(query, [parseInt(limit), parseInt(offset)]);
res.json({
success: true,
data: result.rows,
pagination: {
limit: parseInt(limit),
offset: parseInt(offset),
total: result.rows.length
}
});
} catch (error) {
console.error('Error getting recent Bitbucket webhooks:', error);
res.status(500).json({
success: false,
error: 'Failed to get recent Bitbucket webhooks',
message: error.message
});
}
});
/**
* GET /api/webhooks/gitea/recent
* Get recent Gitea webhooks
*/
router.get('/gitea/recent', async (req, res) => {
try {
const { limit = 50, offset = 0 } = req.query;
const query = `
SELECT id, delivery_id, event_type, action, owner_name, repository_name,
ref, commit_count, created_at
FROM gitea_webhooks
ORDER BY created_at DESC
LIMIT $1 OFFSET $2
`;
const result = await webhookService.db.query(query, [parseInt(limit), parseInt(offset)]);
res.json({
success: true,
data: result.rows,
pagination: {
limit: parseInt(limit),
offset: parseInt(offset),
total: result.rows.length
}
});
} catch (error) {
console.error('Error getting recent Gitea webhooks:', error);
res.status(500).json({
success: false,
error: 'Failed to get recent Gitea webhooks',
message: error.message
});
}
});
/**
* GET /api/webhooks/stats
* Get webhook statistics across all providers
*/
router.get('/stats', async (req, res) => {
try {
const { days_back = 7 } = req.query;
const queries = [
`SELECT 'github' as provider, COUNT(*) as total,
COUNT(*) FILTER (WHERE processed_at IS NOT NULL) as processed
FROM github_webhooks
WHERE created_at >= NOW() - INTERVAL '${parseInt(days_back)} days'`,
`SELECT 'gitlab' as provider, COUNT(*) as total,
COUNT(*) as processed
FROM gitlab_webhooks
WHERE created_at >= NOW() - INTERVAL '${parseInt(days_back)} days'`,
`SELECT 'bitbucket' as provider, COUNT(*) as total,
COUNT(*) as processed
FROM bitbucket_webhooks
WHERE created_at >= NOW() - INTERVAL '${parseInt(days_back)} days'`,
`SELECT 'gitea' as provider, COUNT(*) as total,
COUNT(*) as processed
FROM gitea_webhooks
WHERE created_at >= NOW() - INTERVAL '${parseInt(days_back)} days'`
];
const results = await Promise.all(
queries.map(query => webhookService.db.query(query))
);
const stats = results.map(result => result.rows[0]).filter(row => row.total > 0);
res.json({
success: true,
data: {
period_days: parseInt(days_back),
providers: stats,
total_webhooks: stats.reduce((sum, stat) => sum + parseInt(stat.total), 0),
total_processed: stats.reduce((sum, stat) => sum + parseInt(stat.processed), 0)
}
});
} catch (error) {
console.error('Error getting webhook stats:', error);
res.status(500).json({
success: false,
error: 'Failed to get webhook stats',
message: error.message
});
}
});
/**
* GET /api/webhooks/:provider/:webhookId/payload
* Get full webhook payload by ID
*/
router.get('/:provider/:webhookId/payload', async (req, res) => {
try {
const { provider, webhookId } = req.params;
const tableMap = {
'github': 'github_webhooks',
'gitlab': 'gitlab_webhooks',
'bitbucket': 'bitbucket_webhooks',
'gitea': 'gitea_webhooks'
};
const tableName = tableMap[provider];
if (!tableName) {
return res.status(400).json({
success: false,
error: 'Invalid provider. Must be github, gitlab, bitbucket, or gitea'
});
}
const query = `SELECT payload FROM ${tableName} WHERE id = $1`;
const result = await webhookService.db.query(query, [webhookId]);
if (result.rows.length === 0) {
return res.status(404).json({
success: false,
error: 'Webhook not found'
});
}
res.json({
success: true,
data: result.rows[0].payload
});
} catch (error) {
console.error('Error getting webhook payload:', error);
res.status(500).json({
success: false,
error: 'Failed to get webhook payload',
message: error.message
});
}
});
module.exports = router;

6
services/git-integration/src/routes/github-integration.routes.js
View File

@ -1168,11 +1168,7 @@ router.delete('/repository/:id', async (req, res) => {
// Clean up file storage
await githubService.cleanupRepositoryStorage(id);
// Delete feature mappings first
await database.query(
'DELETE FROM feature_codebase_mappings WHERE repository_id = $1',
[id]
);
// Delete repository record
await database.query(

384
services/git-integration/src/routes/oauth-providers.routes.js Normal file
View File

@ -0,0 +1,384 @@
const express = require('express');
const router = express.Router();
const OAuthTokensModel = require('../models/oauth-tokens.model');
const oauthTokensModel = new OAuthTokensModel();
/**
* POST /api/oauth/gitlab/token
* Store GitLab OAuth token
*/
router.post('/gitlab/token', async (req, res) => {
try {
const {
access_token,
gitlab_username,
gitlab_user_id,
scopes,
expires_at
} = req.body;
if (!access_token || !gitlab_username) {
return res.status(400).json({
success: false,
error: 'access_token and gitlab_username are required'
});
}
const tokenData = {
access_token,
gitlab_username,
gitlab_user_id,
scopes,
expires_at: expires_at ? new Date(expires_at) : null
};
const token = await oauthTokensModel.createGitLabToken(tokenData);
res.json({
success: true,
data: {
id: token.id,
gitlab_username: token.gitlab_username,
scopes: token.scopes,
expires_at: token.expires_at,
created_at: token.created_at
}
});
} catch (error) {
console.error('Error storing GitLab token:', error);
res.status(500).json({
success: false,
error: 'Failed to store GitLab token',
message: error.message
});
}
});
/**
* GET /api/oauth/gitlab/token/:username
* Get GitLab OAuth token by username
*/
router.get('/gitlab/token/:username', async (req, res) => {
try {
const { username } = req.params;
const token = await oauthTokensModel.getGitLabTokenByUsername(username);
if (!token) {
return res.status(404).json({
success: false,
error: 'GitLab token not found'
});
}
res.json({
success: true,
data: {
id: token.id,
gitlab_username: token.gitlab_username,
scopes: token.scopes,
expires_at: token.expires_at,
created_at: token.created_at
}
});
} catch (error) {
console.error('Error getting GitLab token:', error);
res.status(500).json({
success: false,
error: 'Failed to get GitLab token',
message: error.message
});
}
});
/**
* POST /api/oauth/bitbucket/token
* Store Bitbucket OAuth token
*/
router.post('/bitbucket/token', async (req, res) => {
try {
const {
access_token,
bitbucket_username,
bitbucket_user_id,
scopes,
expires_at
} = req.body;
if (!access_token || !bitbucket_username) {
return res.status(400).json({
success: false,
error: 'access_token and bitbucket_username are required'
});
}
const tokenData = {
access_token,
bitbucket_username,
bitbucket_user_id,
scopes,
expires_at: expires_at ? new Date(expires_at) : null
};
const token = await oauthTokensModel.createBitbucketToken(tokenData);
res.json({
success: true,
data: {
id: token.id,
bitbucket_username: token.bitbucket_username,
scopes: token.scopes,
expires_at: token.expires_at,
created_at: token.created_at
}
});
} catch (error) {
console.error('Error storing Bitbucket token:', error);
res.status(500).json({
success: false,
error: 'Failed to store Bitbucket token',
message: error.message
});
}
});
/**
* GET /api/oauth/bitbucket/token/:username
* Get Bitbucket OAuth token by username
*/
router.get('/bitbucket/token/:username', async (req, res) => {
try {
const { username } = req.params;
const token = await oauthTokensModel.getBitbucketTokenByUsername(username);
if (!token) {
return res.status(404).json({
success: false,
error: 'Bitbucket token not found'
});
}
res.json({
success: true,
data: {
id: token.id,
bitbucket_username: token.bitbucket_username,
scopes: token.scopes,
expires_at: token.expires_at,
created_at: token.created_at
}
});
} catch (error) {
console.error('Error getting Bitbucket token:', error);
res.status(500).json({
success: false,
error: 'Failed to get Bitbucket token',
message: error.message
});
}
});
/**
* POST /api/oauth/gitea/token
* Store Gitea OAuth token
*/
router.post('/gitea/token', async (req, res) => {
try {
const {
access_token,
gitea_username,
gitea_user_id,
scopes,
expires_at
} = req.body;
if (!access_token || !gitea_username) {
return res.status(400).json({
success: false,
error: 'access_token and gitea_username are required'
});
}
const tokenData = {
access_token,
gitea_username,
gitea_user_id,
scopes,
expires_at: expires_at ? new Date(expires_at) : null
};
const token = await oauthTokensModel.createGiteaToken(tokenData);
res.json({
success: true,
data: {
id: token.id,
gitea_username: token.gitea_username,
scopes: token.scopes,
expires_at: token.expires_at,
created_at: token.created_at
}
});
} catch (error) {
console.error('Error storing Gitea token:', error);
res.status(500).json({
success: false,
error: 'Failed to store Gitea token',
message: error.message
});
}
});
/**
* GET /api/oauth/gitea/token/:username
* Get Gitea OAuth token by username
*/
router.get('/gitea/token/:username', async (req, res) => {
try {
const { username } = req.params;
const token = await oauthTokensModel.getGiteaTokenByUsername(username);
if (!token) {
return res.status(404).json({
success: false,
error: 'Gitea token not found'
});
}
res.json({
success: true,
data: {
id: token.id,
gitea_username: token.gitea_username,
scopes: token.scopes,
expires_at: token.expires_at,
created_at: token.created_at
}
});
} catch (error) {
console.error('Error getting Gitea token:', error);
res.status(500).json({
success: false,
error: 'Failed to get Gitea token',
message: error.message
});
}
});
/**
* GET /api/oauth/:provider/tokens
* Get all tokens for a provider
*/
router.get('/:provider/tokens', async (req, res) => {
try {
const { provider } = req.params;
if (!['gitlab', 'bitbucket', 'gitea'].includes(provider)) {
return res.status(400).json({
success: false,
error: 'Invalid provider. Must be gitlab, bitbucket, or gitea'
});
}
const tokens = await oauthTokensModel.getAllTokensByProvider(provider);
// Remove sensitive access_token from response
const sanitizedTokens = tokens.map(token => {
const { access_token, ...safeToken } = token;
return safeToken;
});
res.json({
success: true,
data: sanitizedTokens
});
} catch (error) {
console.error('Error getting provider tokens:', error);
res.status(500).json({
success: false,
error: 'Failed to get provider tokens',
message: error.message
});
}
});
/**
* DELETE /api/oauth/expired-tokens
* Clean up expired tokens
*/
router.delete('/expired-tokens', async (req, res) => {
try {
const deletedCount = await oauthTokensModel.deleteExpiredTokens();
res.json({
success: true,
data: {
deleted_count: deletedCount
},
message: `Deleted ${deletedCount} expired tokens`
});
} catch (error) {
console.error('Error deleting expired tokens:', error);
res.status(500).json({
success: false,
error: 'Failed to delete expired tokens',
message: error.message
});
}
});
/**
* PUT /api/oauth/gitlab/token/:tokenId
* Update GitLab OAuth token
*/
router.put('/gitlab/token/:tokenId', async (req, res) => {
try {
const { tokenId } = req.params;
const { access_token, scopes, expires_at } = req.body;
if (!access_token) {
return res.status(400).json({
success: false,
error: 'access_token is required'
});
}
const tokenData = {
access_token,
scopes,
expires_at: expires_at ? new Date(expires_at) : null
};
const updatedToken = await oauthTokensModel.updateGitLabToken(tokenId, tokenData);
if (!updatedToken) {
return res.status(404).json({
success: false,
error: 'Token not found'
});
}
res.json({
success: true,
data: {
id: updatedToken.id,
gitlab_username: updatedToken.gitlab_username,
scopes: updatedToken.scopes,
expires_at: updatedToken.expires_at,
updated_at: updatedToken.updated_at
}
});
} catch (error) {
console.error('Error updating GitLab token:', error);
res.status(500).json({
success: false,
error: 'Failed to update GitLab token',
message: error.message
});
}
});
module.exports = router;

83
services/git-integration/src/routes/vcs.routes.js
View File

@ -156,10 +156,7 @@ router.post('/:provider/attach-repository', async (req, res) => {
mappingValues.push(`(uuid_generate_v4(), $${i++}, $${i++}, $${i++}, $${i++})`);
params.push(feature.id, repositoryRecord.id, '[]', '{}');
}
await database.query(
`INSERT INTO feature_codebase_mappings (id, feature_id, repository_id, code_paths, code_snippets) VALUES ${mappingValues.join(', ')}`,
params
);
}
const storageInfo = await (async () => {
@ -209,7 +206,7 @@ router.post('/:provider/webhook', async (req, res) => {
}
// Signature verification
const rawBody = JSON.stringify(payload);
const rawBody = req.rawBody ? req.rawBody : Buffer.from(JSON.stringify(payload));
const verifySignature = () => {
try {
if (providerKey === 'gitlab') {
@ -220,12 +217,19 @@ router.post('/:provider/webhook', async (req, res) => {
}
if (providerKey === 'gitea') {
const crypto = require('crypto');
const provided = req.headers['x-gitea-signature'];
const providedHeader = req.headers['x-gitea-signature'] || req.headers['x-gogs-signature'] || req.headers['x-hub-signature-256'];
const secret = process.env.GITEA_WEBHOOK_SECRET;
if (!secret) return true;
if (!provided) return false;
if (!providedHeader) return false;
let provided = String(providedHeader);
if (provided.startsWith('sha256=')) provided = provided.slice('sha256='.length);
const expected = crypto.createHmac('sha256', secret).update(rawBody).digest('hex');
return crypto.timingSafeEqual(Buffer.from(expected, 'hex'), Buffer.from(provided, 'hex'));
try {
return crypto.timingSafeEqual(Buffer.from(expected, 'hex'), Buffer.from(provided, 'hex'));
} catch (_) {
return false;
}
}
if (providerKey === 'bitbucket') {
// Bitbucket Cloud webhooks typically have no shared secret by default
@ -241,6 +245,14 @@ router.post('/:provider/webhook', async (req, res) => {
return res.status(401).json({ success: false, message: 'Invalid webhook signature' });
}
if (providerKey === 'bitbucket') {
console.log('🔔 Bitbucket webhook received:', {
eventKey: req.headers['x-event-key'],
requestId: req.headers['x-request-id'],
userAgent: req.headers['user-agent'],
payloadSize: rawBody?.length || 0
});
}
// Process webhook event using comprehensive service
const eventType = extractEventType(providerKey, payload);
await vcsWebhookService.processWebhookEvent(providerKey, eventType, payload);
@ -378,10 +390,8 @@ router.get('/:provider/repository/:id/file-content', async (req, res) => {
return res.status(400).json({ success: false, message: 'File path is required' });
}
const query = `
SELECT rf.*, rfc.content_text, rfc.content_preview, rfc.language_detected,
rfc.line_count, rfc.char_count
SELECT rf.*t
FROM repository_files rf
LEFT JOIN repository_file_contents rfc ON rf.id = rfc.file_id
WHERE rf.repository_id = $1 AND rf.relative_path = $2
`;
const result = await database.query(query, [id, file_path]);
@ -449,7 +459,6 @@ router.delete('/:provider/repository/:id', async (req, res) => {
}
const repository = getResult.rows[0];
await fileStorageService.cleanupRepositoryStorage(id);
await database.query('DELETE FROM feature_codebase_mappings WHERE repository_id = $1', [id]);
await database.query('DELETE FROM github_repositories WHERE id = $1', [id]);
res.json({ success: true, message: 'Repository removed successfully', data: { removed_repository: repository.repository_name, template_id: repository.template_id } });
} catch (error) {
@ -459,7 +468,7 @@ router.delete('/:provider/repository/:id', async (req, res) => {
});
// OAuth placeholders (start/callback) per provider for future implementation
router.get('/:provider/auth/start', (req, res) => {
router.get('/:provider/auth/start', async (req, res) => {
try {
const providerKey = (req.params.provider || '').toLowerCase();
const oauth = getOAuthService(providerKey);
@ -479,14 +488,29 @@ router.get('/:provider/auth/callback', (req, res) => {
const code = req.query.code;
const error = req.query.error;
const errorDescription = req.query.error_description;
console.log(`🔄 [VCS OAUTH] Callback received for ${providerKey}:`, {
hasCode: !!code,
hasError: !!error,
code: code?.substring(0, 10) + '...',
error,
errorDescription
});
const oauth = getOAuthService(providerKey);
if (!oauth) return res.status(400).json({ success: false, message: 'Unsupported provider or OAuth not available' });
if (!code) {
// Surface upstream provider error details if present
if (error || errorDescription) {
return res.status(400).json({ success: false, message: 'OAuth error from provider', provider: providerKey, error: error || 'unknown_error', error_description: errorDescription || null, query: req.query });
console.error(`❌ [VCS OAUTH] Provider error for ${providerKey}:`, { error, errorDescription });
return res.status(400).json({
success: false,
message: 'OAuth error from provider',
provider: providerKey,
error: error || 'unknown_error',
error_description: errorDescription || null,
query: req.query
});
}
return res.status(400).json({ success: false, message: 'Missing code' });
return res.status(400).json({ success: false, message: 'Missing authorization code' });
}
const accessToken = await oauth.exchangeCodeForToken(code);
const user = await oauth.getUserInfo(accessToken);
@ -504,7 +528,34 @@ router.get('/:provider/auth/callback', (req, res) => {
const tokenRecord = await oauth.storeToken(accessToken, user, userId || null);
res.json({ success: true, provider: providerKey, user, token: { id: tokenRecord.id || null } });
} catch (e) {
res.status(500).json({ success: false, message: e.message || 'OAuth callback failed' });
console.error(`❌ [VCS OAUTH] Callback error for ${req.params.provider}:`, e);
// Provide more specific error messages
let errorMessage = e.message || 'OAuth callback failed';
let statusCode = 500;
if (e.message.includes('not configured')) {
statusCode = 500;
errorMessage = `OAuth configuration error: ${e.message}`;
} else if (e.message.includes('timeout')) {
statusCode = 504;
errorMessage = `OAuth timeout: ${e.message}`;
} else if (e.message.includes('network error') || e.message.includes('Cannot connect')) {
statusCode = 502;
errorMessage = `Network error: ${e.message}`;
} else if (e.message.includes('HTTP error')) {
statusCode = 502;
errorMessage = `OAuth provider error: ${e.message}`;
}
res.status(statusCode).json({
success: false,
message: errorMessage,
provider: req.params.provider,
error: e.message,
details: process.env.NODE_ENV === 'development' ? e.stack : undefined
});
}
})();
});

16
services/git-integration/src/routes/webhook.routes.js
View File

@ -45,21 +45,7 @@ router.post('/webhook', async (req, res) => {
await webhookService.processWebhookEvent(eventType, payloadWithDelivery);
}
// Log the webhook event
await webhookService.logWebhookEvent(
eventType || 'unknown',
req.body.action || 'unknown',
req.body.repository?.full_name || 'unknown',
{
delivery_id: deliveryId,
event_type: eventType,
action: req.body.action,
repository: req.body.repository?.full_name,
sender: req.body.sender?.login
},
deliveryId,
payloadWithDelivery
);
res.status(200).json({
success: true,

5
services/git-integration/src/services/bitbucket-oauth.js
View File

@ -10,12 +10,13 @@ class BitbucketOAuthService {
getAuthUrl(state) {
if (!this.clientId) throw new Error('Bitbucket OAuth not configured');
const scopes = process.env.BITBUCKET_OAUTH_SCOPES || 'repository account';
const params = new URLSearchParams({
client_id: this.clientId,
response_type: 'code',
state,
// Bitbucket Cloud uses 'repository' for read access; 'repository:write' for write
scope: 'repository account',
scope: scopes,
redirect_uri: this.redirectUri
});
return `https://bitbucket.org/site/oauth2/authorize?${params.toString()}`;
@ -48,7 +49,7 @@ class BitbucketOAuthService {
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (id) DO UPDATE SET access_token = EXCLUDED.access_token, bitbucket_username = EXCLUDED.bitbucket_username, bitbucket_user_id = EXCLUDED.bitbucket_user_id, scopes = EXCLUDED.scopes, expires_at = EXCLUDED.expires_at, updated_at = NOW()
RETURNING *`,
[accessToken, user.username || user.display_name, user.uuid || null, JSON.stringify(['repository:read','account']), null]
[accessToken, user.username || user.display_name, user.uuid || null, JSON.stringify(['repository:admin','webhook','account']), null]
);
return result.rows[0];
}

239
services/git-integration/src/services/commit-tracking.service.js Normal file
View File

@ -0,0 +1,239 @@
const CommitDetailsModel = require('../models/commit-details.model');
const CommitFilesModel = require('../models/commit-files.model');
const DiffStorageModel = require('../models/diff-storage.model');
class CommitTrackingService {
constructor() {
this.commitDetailsModel = new CommitDetailsModel();
this.commitFilesModel = new CommitFilesModel();
this.diffStorageModel = new DiffStorageModel();
}
/**
* Process a complete commit with its file changes
* @param {Object} commitData - Commit information
* @param {Array} fileChanges - Array of file changes
* @returns {Object} Processed commit with files
*/
async processCommitWithFiles(commitData, fileChanges = []) {
try {
// Create commit detail record
const commit = await this.commitDetailsModel.createCommitDetail(commitData);
// Create file change records if provided
let files = [];
if (fileChanges && fileChanges.length > 0) {
files = await this.commitFilesModel.createMultipleCommitFiles(commit.id, fileChanges);
// Queue diff processing for this commit
await this.queueDiffProcessing(commit, commitData.repository_id);
}
return {
commit,
files,
total_files: files.length
};
} catch (error) {
console.error('Error processing commit with files:', error);
throw error;
}
}
/**
* Queue diff processing for a commit
* @param {Object} commit - Commit details
* @param {string} repository_id - Repository ID
*/
async queueDiffProcessing(commit, repository_id) {
try {
const queueData = {
commit_id: commit.id,
repository_id: repository_id,
to_sha: commit.commit_sha,
repo_local_path: `/tmp/repos/${repository_id}`, // This should be configurable
priority: 1
};
await this.diffStorageModel.addToProcessingQueue(queueData);
console.log(`Queued diff processing for commit ${commit.commit_sha}`);
} catch (error) {
console.error('Error queuing diff processing:', error);
// Don't throw - this is not critical for commit processing
}
}
/**
* Get commit history for a repository with file changes
* @param {string} repository_id - Repository ID
* @param {Object} options - Query options
* @returns {Array} Commit history with file counts
*/
async getCommitHistory(repository_id, options = {}) {
const { limit = 50, offset = 0, include_files = false } = options;
try {
const commits = await this.commitDetailsModel.getCommitsByRepository(
repository_id,
limit,
offset
);
if (include_files) {
// Fetch file changes for each commit
for (const commit of commits) {
const files = await this.commitFilesModel.getFilesByCommit(commit.id);
commit.files = files;
commit.file_count = files.length;
}
} else {
// Just get file counts
for (const commit of commits) {
const files = await this.commitFilesModel.getFilesByCommit(commit.id);
commit.file_count = files.length;
}
}
return commits;
} catch (error) {
console.error('Error getting commit history:', error);
throw error;
}
}
/**
* Get file change history for a specific file path
* @param {string} repository_id - Repository ID
* @param {string} file_path - File path
* @returns {Array} File change history
*/
async getFileHistory(repository_id, file_path) {
try {
return await this.commitFilesModel.getFileChangesByPath(repository_id, file_path);
} catch (error) {
console.error('Error getting file history:', error);
throw error;
}
}
/**
* Process webhook commit data
* @param {Object} webhookPayload - Webhook payload
* @param {string} repository_id - Repository ID
* @returns {Array} Processed commits
*/
async processWebhookCommits(webhookPayload, repository_id) {
try {
const commits = webhookPayload.commits || [];
const processedCommits = [];
for (const commitData of commits) {
const processedCommit = await this.processCommitFromWebhook(commitData, repository_id);
processedCommits.push(processedCommit);
}
return processedCommits;
} catch (error) {
console.error('Error processing webhook commits:', error);
throw error;
}
}
/**
* Process a single commit from webhook data
* @param {Object} commitData - Commit data from webhook
* @param {string} repository_id - Repository ID
* @returns {Object} Processed commit
*/
async processCommitFromWebhook(commitData, repository_id) {
try {
const commit = {
repository_id,
commit_sha: commitData.id,
author_name: commitData.author?.name,
author_email: commitData.author?.email,
message: commitData.message,
url: commitData.url,
committed_at: new Date(commitData.timestamp)
};
// Extract file changes
const fileChanges = [];
// Added files
if (commitData.added) {
commitData.added.forEach(filePath => {
fileChanges.push({
change_type: 'added',
file_path: filePath
});
});
}
// Modified files
if (commitData.modified) {
commitData.modified.forEach(filePath => {
fileChanges.push({
change_type: 'modified',
file_path: filePath
});
});
}
// Removed files
if (commitData.removed) {
commitData.removed.forEach(filePath => {
fileChanges.push({
change_type: 'removed',
file_path: filePath
});
});
}
return await this.processCommitWithFiles(commit, fileChanges);
} catch (error) {
console.error('Error processing commit from webhook:', error);
throw error;
}
}
/**
* Get repository statistics
* @param {string} repository_id - Repository ID
* @returns {Object} Repository statistics
*/
async getRepositoryStats(repository_id) {
try {
const commits = await this.commitDetailsModel.getCommitsByRepository(repository_id, 1000);
const recentFiles = await this.commitFilesModel.getFilesByRepository(repository_id, 100);
// Calculate statistics
const totalCommits = commits.length;
const uniqueAuthors = new Set(commits.map(c => c.author_email)).size;
const recentActivity = commits.filter(c => {
const commitDate = new Date(c.committed_at);
const thirtyDaysAgo = new Date(Date.now() - 30 * 24 * 60 * 60 * 1000);
return commitDate > thirtyDaysAgo;
}).length;
// File change statistics
const fileStats = recentFiles.reduce((acc, file) => {
acc[file.change_type] = (acc[file.change_type] || 0) + 1;
return acc;
}, {});
return {
total_commits: totalCommits,
unique_authors: uniqueAuthors,
recent_activity_30_days: recentActivity,
file_changes: fileStats,
last_commit: commits[0] || null
};
} catch (error) {
console.error('Error getting repository stats:', error);
throw error;
}
}
}
module.exports = CommitTrackingService;

421
services/git-integration/src/services/diff-processing.service.js Normal file
View File

@ -0,0 +1,421 @@
// services/diff-processing.service.js
const fs = require('fs');
const path = require('path');
const zlib = require('zlib');
const database = require('../config/database');
const GitRepoService = require('./git-repo.service');
class DiffProcessingService {
constructor() {
this.gitService = new GitRepoService();
this.baseDir = process.env.ATTACHED_REPOS_DIR || '/tmp/attached-repos';
// Allow overriding diff storage root via env; fallback to <ATTACHED_REPOS_DIR>/diffs
const envDiffDir = process.env.DIFF_STORAGE_DIR && process.env.DIFF_STORAGE_DIR.trim().length > 0
? process.env.DIFF_STORAGE_DIR
: null;
this.diffStorageDir = envDiffDir || path.join(this.baseDir, 'diffs');
// Size threshold: <= 50KB store in DB; > 50KB store on disk
this.SIZE_THRESHOLDS = {
SMALL: 50 * 1024
};
this.ensureDiffStorageDir();
}
ensureDiffStorageDir() {
if (!fs.existsSync(this.diffStorageDir)) {
fs.mkdirSync(this.diffStorageDir, { recursive: true });
}
}
// Main method to process diffs for a commit
async processCommitDiffs(commitId, repositoryId, repoPath, fromSha, toSha) {
console.log(`🔄 Processing diffs for commit ${commitId}`);
try {
// Get the diff content
const diffContent = await this.gitService.getDiff(repoPath, fromSha, toSha);
if (!diffContent || diffContent.trim().length === 0) {
console.log(`⚠️ No diff content found for commit ${commitId}`);
return { success: false, reason: 'No diff content' };
}
// Parse the diff to extract individual file changes
const fileDiffs = this.parseDiffContent(diffContent);
console.log(`📄 Found ${fileDiffs.length} file changes in diff`);
// Process each file diff
const results = [];
for (const fileDiff of fileDiffs) {
try {
const result = await this.processFileDiff(commitId, fileDiff);
results.push(result);
} catch (error) {
console.error(`❌ Failed to process diff for file ${fileDiff.filePath}:`, error.message);
results.push({
success: false,
filePath: fileDiff.filePath,
error: error.message
});
}
}
// Update processing statistics
await this.updateDiffStatistics(repositoryId, results);
return {
success: true,
processedFiles: results.length,
results: results
};
} catch (error) {
console.error(`❌ Failed to process commit diffs:`, error.message);
return { success: false, error: error.message };
}
}
// Parse diff content into individual file diffs
parseDiffContent(diffContent) {
const fileDiffs = [];
const lines = diffContent.split('\n');
let currentFileDiff = null;
let currentDiffBody = [];
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
// Check for file header (diff --git)
if (line.startsWith('diff --git')) {
// Save previous file diff if exists
if (currentFileDiff) {
currentFileDiff.diffBody = currentDiffBody.join('\n');
// Infer accurate change type and canonical paths from diff body
this.inferChangeTypeAndPaths(currentFileDiff);
fileDiffs.push(currentFileDiff);
}
// Start new file diff
currentFileDiff = this.parseFileHeader(line);
currentDiffBody = [line];
} else if (currentFileDiff) {
currentDiffBody.push(line);
}
}
// Don't forget the last file diff
if (currentFileDiff) {
currentFileDiff.diffBody = currentDiffBody.join('\n');
this.inferChangeTypeAndPaths(currentFileDiff);
fileDiffs.push(currentFileDiff);
}
return fileDiffs;
}
// Parse file header from diff line
parseFileHeader(diffLine) {
// Example: diff --git a/src/app.js b/src/app.js
const match = diffLine.match(/diff --git a\/(.+) b\/(.+)/);
if (!match) {
throw new Error(`Invalid diff header: ${diffLine}`);
}
const oldPath = match[1];
const newPath = match[2];
// Determine change type
let changeType = 'modified';
if (oldPath === '/dev/null') {
changeType = 'added';
} else if (newPath === '/dev/null') {
changeType = 'deleted';
} else if (oldPath !== newPath) {
changeType = 'renamed';
}
return {
filePath: newPath === '/dev/null' ? oldPath : newPath,
oldPath: oldPath === '/dev/null' ? null : oldPath,
newPath: newPath === '/dev/null' ? null : newPath,
changeType: changeType,
diffHeader: diffLine
};
}
// Inspect the diff body to accurately infer change type and canonical paths
inferChangeTypeAndPaths(fileDiff) {
if (!fileDiff || !fileDiff.diffBody) return;
const lines = fileDiff.diffBody.split('\n');
let headerOld = null;
let headerNew = null;
let sawNewFile = false;
let sawDeletedFile = false;
let sawRenameFrom = null;
let sawRenameTo = null;
for (const l of lines) {
if (l.startsWith('new file mode')) sawNewFile = true;
if (l.startsWith('deleted file mode')) sawDeletedFile = true;
if (l.startsWith('rename from ')) sawRenameFrom = l.replace('rename from ', '').trim();
if (l.startsWith('rename to ')) sawRenameTo = l.replace('rename to ', '').trim();
if (l.startsWith('--- ')) headerOld = l.substring(4).trim(); // e.g., 'a/path' or '/dev/null'
if (l.startsWith('+++ ')) headerNew = l.substring(4).trim(); // e.g., 'b/path' or '/dev/null'
// Stop early after we've collected the header markers
if (headerOld && headerNew && (sawNewFile || sawDeletedFile || sawRenameFrom || sawRenameTo)) break;
}
// Normalize paths like 'a/path' or 'b/path'
const normalize = (p) => {
if (!p) return null;
if (p === '/dev/null') return '/dev/null';
// strip leading a/ or b/
return p.replace(/^a\//, '').replace(/^b\//, '');
};
const oldPath = normalize(headerOld);
const newPath = normalize(headerNew);
// Decide change type priority: rename > added > deleted > modified
if (sawRenameFrom || sawRenameTo) {
fileDiff.changeType = 'renamed';
fileDiff.oldPath = sawRenameFrom || oldPath || fileDiff.oldPath || null;
fileDiff.newPath = sawRenameTo || newPath || fileDiff.newPath || null;
fileDiff.filePath = fileDiff.newPath || fileDiff.filePath;
return;
}
if (sawNewFile || oldPath === '/dev/null') {
fileDiff.changeType = 'added';
fileDiff.oldPath = null;
fileDiff.newPath = newPath || fileDiff.newPath || fileDiff.filePath;
fileDiff.filePath = fileDiff.newPath;
return;
}
if (sawDeletedFile || newPath === '/dev/null') {
fileDiff.changeType = 'deleted';
fileDiff.newPath = null;
fileDiff.oldPath = oldPath || fileDiff.oldPath || fileDiff.filePath;
fileDiff.filePath = fileDiff.oldPath;
return;
}
// Default to modified; refine filePath if headers present
fileDiff.changeType = 'modified';
if (newPath && newPath !== '/dev/null') {
fileDiff.filePath = newPath;
}
if (!fileDiff.oldPath && oldPath && oldPath !== '/dev/null') fileDiff.oldPath = oldPath;
if (!fileDiff.newPath && newPath && newPath !== '/dev/null') fileDiff.newPath = newPath;
}
// Process individual file diff
async processFileDiff(commitId, fileDiff) {
const diffSize = Buffer.byteLength(fileDiff.diffBody, 'utf8');
const storageType = 'external';
console.log(`📊 File ${fileDiff.filePath}: ${diffSize} bytes, storage: ${storageType}`);
// Get file change record
const fileChangeQuery = `
SELECT rcf.id
FROM repository_commit_files rcf
JOIN repository_commit_details rcd ON rcf.commit_id = rcd.id
WHERE rcd.id = $1 AND rcf.file_path = $2
`;
const fileChangeResult = await database.query(fileChangeQuery, [commitId, fileDiff.filePath]);
if (fileChangeResult.rows.length === 0) {
throw new Error(`File change record not found for ${fileDiff.filePath}`);
}
const fileChangeId = fileChangeResult.rows[0].id;
// Store diff based on size
const diffContentId = await this.storeExternally(commitId, fileChangeId, fileDiff, diffSize);
return {
success: true,
filePath: fileDiff.filePath,
changeType: fileDiff.changeType,
diffSize: diffSize,
storageType: storageType,
diffContentId: diffContentId
};
}
// Determine storage type (always external)
determineStorageType(size) { return 'external'; }
// Removed DB storage path (all diffs stored externally)
// Remove compression path; no longer used
// Store large diff externally
async storeExternally(commitId, fileChangeId, fileDiff, diffSize) {
// Create directory structure: diffs/repo_id/commit_sha/
const commitDir = path.join(this.diffStorageDir, commitId);
if (!fs.existsSync(commitDir)) {
fs.mkdirSync(commitDir, { recursive: true });
}
// Create safe filename
const safeFileName = fileDiff.filePath.replace(/[^a-zA-Z0-9._-]/g, '_') + '.diff';
const filePath = path.join(commitDir, safeFileName);
// Write diff to file
fs.writeFileSync(filePath, fileDiff.diffBody, 'utf8');
const query = `
INSERT INTO diff_contents (
commit_id, file_change_id, diff_header, diff_size_bytes,
storage_type, external_storage_path, external_storage_provider,
file_path, change_type, processing_status
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
RETURNING id
`;
const result = await database.query(query, [
commitId, fileChangeId, fileDiff.diffHeader, diffSize,
'external', filePath, 'local',
fileDiff.filePath, fileDiff.changeType, 'processed'
]);
return result.rows[0].id;
}
// Update diff processing statistics
async updateDiffStatistics(repositoryId, results) {
const successful = results.filter(r => r.success);
const failed = results.filter(r => !r.success);
const totalSize = successful.reduce((sum, r) => sum + (r.diffSize || 0), 0);
const avgSize = successful.length > 0 ? totalSize / successful.length : 0;
const externalCount = successful.filter(r => r.storageType === 'external').length;
const query = `
INSERT INTO diff_statistics (
repository_id, period_start, period_end,
total_commits, total_files_changed, total_diffs_processed,
total_diff_size_bytes, avg_diff_size_bytes, max_diff_size_bytes,
diffs_stored_external, failed_processing_count
) VALUES ($1, NOW() - INTERVAL '1 hour', NOW(),
1, $2, $3, $4, $5, $6, $7, $8)
`;
await database.query(query, [
repositoryId, successful.length, successful.length,
totalSize, avgSize, Math.max(...successful.map(r => r.diffSize || 0), 0),
externalCount, failed.length
]);
}
// Queue diff processing for background processing
async queueDiffProcessing(commitId, repositoryId, repoPath, fromSha, toSha, priority = 0) {
const query = `
INSERT INTO diff_processing_queue (
commit_id, repository_id, from_sha, to_sha, repo_local_path, priority
) VALUES ($1, $2, $3, $4, $5, $6)
RETURNING id
`;
const result = await database.query(query, [
commitId, repositoryId, fromSha, toSha, repoPath, priority
]);
return result.rows[0].id;
}
// Get diff content for retrieval
async getDiffContent(diffContentId) {
const query = `
SELECT storage_type, external_storage_path, file_path, change_type
FROM diff_contents
WHERE id = $1
`;
const result = await database.query(query, [diffContentId]);
if (result.rows.length === 0) {
throw new Error(`Diff content not found: ${diffContentId}`);
}
const diff = result.rows[0];
// We only support external storage now
return fs.readFileSync(diff.external_storage_path, 'utf8');
}
// Get diffs for a commit
async getCommitDiffs(commitId) {
const query = `
SELECT dc.*, rcf.change_type as file_change_type
FROM diff_contents dc
JOIN repository_commit_files rcf ON dc.file_change_id = rcf.id
WHERE dc.commit_id = $1
ORDER BY dc.file_path
`;
const result = await database.query(query, [commitId]);
return result.rows;
}
// Get repository diff statistics
async getRepositoryDiffStats(repositoryId, daysBack = 30) {
const query = `
SELECT * FROM get_repository_diff_stats($1, $2)
`;
const result = await database.query(query, [repositoryId, daysBack]);
return result.rows[0];
}
// Clean up old external diff files
async cleanupOldDiffs(daysBack = 30) {
const cutoffDate = new Date();
cutoffDate.setDate(cutoffDate.getDate() - daysBack);
// Get old external diffs
const query = `
SELECT external_storage_path
FROM diff_contents
WHERE storage_type = 'external'
AND created_at < $1
`;
const result = await database.query(query, [cutoffDate]);
// Delete files
let deletedCount = 0;
for (const row of result.rows) {
try {
if (fs.existsSync(row.external_storage_path)) {
fs.unlinkSync(row.external_storage_path);
deletedCount++;
}
} catch (error) {
console.warn(`Failed to delete diff file ${row.external_storage_path}:`, error.message);
}
}
// Delete database records
const deleteQuery = `
DELETE FROM diff_contents
WHERE storage_type = 'external'
AND created_at < $1
`;
await database.query(deleteQuery, [cutoffDate]);
return deletedCount;
}
}
module.exports = DiffProcessingService;

336
services/git-integration/src/services/enhanced-diff-processing.service.js Normal file
View File

@ -0,0 +1,336 @@
const fs = require('fs').promises;
const path = require('path');
const { execSync } = require('child_process');
const DiffStorageModel = require('../models/diff-storage.model');
class EnhancedDiffProcessingService {
constructor() {
this.diffStorageModel = new DiffStorageModel();
this.storageBasePath = process.env.DIFF_STORAGE_PATH || '/tmp/diffs';
this.maxDiffSize = parseInt(process.env.MAX_DIFF_SIZE_BYTES) || 10 * 1024 * 1024; // 10MB default
}
/**
* Initialize the diff processing service
*/
async initialize() {
try {
await fs.mkdir(this.storageBasePath, { recursive: true });
console.log(`Diff storage initialized at: ${this.storageBasePath}`);
} catch (error) {
console.error('Error initializing diff storage:', error);
throw error;
}
}
/**
* Process the next item in the diff processing queue
* @returns {Object|null} Processing result or null if no items
*/
async processNextQueueItem() {
try {
const queueItem = await this.diffStorageModel.getNextQueueItem();
if (!queueItem) {
return null;
}
console.log(`Processing diff queue item: ${queueItem.id}`);
// Update status to processing
await this.diffStorageModel.updateQueueStatus(queueItem.id, 'processing');
try {
const result = await this.processDiffForCommit(queueItem);
// Update status to completed
await this.diffStorageModel.updateQueueStatus(queueItem.id, 'completed');
return result;
} catch (error) {
console.error(`Error processing diff queue item ${queueItem.id}:`, error);
// Increment retry count
const updatedItem = await this.diffStorageModel.incrementRetryCount(queueItem.id);
// Check if we should retry or mark as failed
if (updatedItem.retry_count >= updatedItem.max_retries) {
await this.diffStorageModel.updateQueueStatus(
queueItem.id,
'failed',
error.message
);
} else {
await this.diffStorageModel.updateQueueStatus(queueItem.id, 'pending');
}
throw error;
}
} catch (error) {
console.error('Error processing queue item:', error);
throw error;
}
}
/**
* Process diff for a specific commit
* @param {Object} queueItem - Queue item data
* @returns {Object} Processing result
*/
async processDiffForCommit(queueItem) {
const { commit_id, repository_id, from_sha, to_sha, repo_local_path } = queueItem;
try {
// Generate diff using git
const diffOutput = await this.generateGitDiff(repo_local_path, from_sha, to_sha);
if (!diffOutput || diffOutput.trim().length === 0) {
console.log(`No diff output for commit ${to_sha}`);
return { processed_files: 0, total_size: 0 };
}
// Parse diff output into individual file diffs
const fileDiffs = this.parseDiffOutput(diffOutput);
const processedFiles = [];
let totalSize = 0;
for (const fileDiff of fileDiffs) {
try {
const diffContent = await this.storeDiffContent(
commit_id,
repository_id,
fileDiff
);
processedFiles.push(diffContent);
totalSize += diffContent.diff_size_bytes;
} catch (error) {
console.error(`Error storing diff for file ${fileDiff.file_path}:`, error);
// Continue with other files
}
}
console.log(`Processed ${processedFiles.length} file diffs, total size: ${totalSize} bytes`);
return {
processed_files: processedFiles.length,
total_size: totalSize,
files: processedFiles
};
} catch (error) {
console.error('Error processing diff for commit:', error);
throw error;
}
}
/**
* Generate git diff using command line
* @param {string} repoPath - Repository path
* @param {string} fromSha - From commit SHA
* @param {string} toSha - To commit SHA
* @returns {string} Diff output
*/
async generateGitDiff(repoPath, fromSha, toSha) {
try {
const command = fromSha
? `git -C "${repoPath}" diff ${fromSha}..${toSha}`
: `git -C "${repoPath}" show --format="" ${toSha}`;
const diffOutput = execSync(command, {
encoding: 'utf8',
maxBuffer: this.maxDiffSize
});
return diffOutput;
} catch (error) {
if (error.code === 'ENOENT') {
throw new Error(`Repository not found at path: ${repoPath}`);
}
throw new Error(`Git diff failed: ${error.message}`);
}
}
/**
* Parse diff output into individual file diffs
* @param {string} diffOutput - Raw diff output
* @returns {Array} Array of file diff objects
*/
parseDiffOutput(diffOutput) {
const fileDiffs = [];
const lines = diffOutput.split('\n');
let currentFile = null;
let currentDiff = [];
for (const line of lines) {
if (line.startsWith('diff --git ')) {
// Save previous file diff if exists
if (currentFile && currentDiff.length > 0) {
fileDiffs.push({
...currentFile,
diff_content: currentDiff.join('\n')
});
}
// Start new file diff
const match = line.match(/diff --git a\/(.+) b\/(.+)/);
if (match) {
currentFile = {
file_path: match[2],
diff_header: line,
change_type: 'modified' // Default, will be refined below
};
currentDiff = [line];
}
} else if (line.startsWith('new file mode')) {
if (currentFile) currentFile.change_type = 'added';
currentDiff.push(line);
} else if (line.startsWith('deleted file mode')) {
if (currentFile) currentFile.change_type = 'deleted';
currentDiff.push(line);
} else if (line.startsWith('rename from')) {
if (currentFile) currentFile.change_type = 'renamed';
currentDiff.push(line);
} else {
currentDiff.push(line);
}
}
// Don't forget the last file
if (currentFile && currentDiff.length > 0) {
fileDiffs.push({
...currentFile,
diff_content: currentDiff.join('\n')
});
}
return fileDiffs;
}
/**
* Store diff content to external storage
* @param {string} commit_id - Commit ID
* @param {string} repository_id - Repository ID
* @param {Object} fileDiff - File diff data
* @returns {Object} Stored diff content record
*/
async storeDiffContent(commit_id, repository_id, fileDiff) {
try {
const diffSizeBytes = Buffer.byteLength(fileDiff.diff_content, 'utf8');
// Generate storage path
const fileName = `${commit_id}_${this.sanitizeFileName(fileDiff.file_path)}.diff`;
const storagePath = path.join(this.storageBasePath, repository_id, fileName);
// Ensure directory exists
await fs.mkdir(path.dirname(storagePath), { recursive: true });
// Write diff content to file
await fs.writeFile(storagePath, fileDiff.diff_content, 'utf8');
// Create database record
const diffData = {
commit_id,
diff_header: fileDiff.diff_header,
diff_size_bytes: diffSizeBytes,
storage_type: 'external',
external_storage_path: storagePath,
external_storage_provider: 'local',
file_path: fileDiff.file_path,
change_type: fileDiff.change_type,
processing_status: 'processed'
};
const diffContent = await this.diffStorageModel.createDiffContent(diffData);
console.log(`Stored diff for ${fileDiff.file_path} (${diffSizeBytes} bytes)`);
return diffContent;
} catch (error) {
console.error('Error storing diff content:', error);
throw error;
}
}
/**
* Retrieve diff content from storage
* @param {string} diff_id - Diff content ID
* @returns {string} Diff content
*/
async retrieveDiffContent(diff_id) {
try {
const diffRecord = await this.diffStorageModel.getDiffById(diff_id);
if (!diffRecord) {
throw new Error(`Diff record not found: ${diff_id}`);
}
if (diffRecord.storage_type === 'external') {
const content = await fs.readFile(diffRecord.external_storage_path, 'utf8');
return content;
}
throw new Error(`Unsupported storage type: ${diffRecord.storage_type}`);
} catch (error) {
console.error('Error retrieving diff content:', error);
throw error;
}
}
/**
* Start background processing of diff queue
* @param {number} intervalMs - Processing interval in milliseconds
*/
startBackgroundProcessing(intervalMs = 30000) {
console.log(`Starting background diff processing (interval: ${intervalMs}ms)`);
const processQueue = async () => {
try {
const result = await this.processNextQueueItem();
if (result) {
console.log(`Processed diff queue item: ${result.processed_files} files`);
}
} catch (error) {
console.error('Background diff processing error:', error);
}
};
// Process immediately and then on interval
processQueue();
setInterval(processQueue, intervalMs);
}
/**
* Clean up old diff files and queue entries
* @param {number} daysBack - Days to keep
* @returns {Object} Cleanup results
*/
async cleanupOldData(daysBack = 30) {
try {
// Clean up queue entries
const deletedQueueItems = await this.diffStorageModel.cleanupOldQueueEntries(7);
// TODO: Clean up old diff files from storage
// This would require querying diff_contents for old records and deleting files
console.log(`Cleanup completed: ${deletedQueueItems} queue items deleted`);
return {
deleted_queue_items: deletedQueueItems
};
} catch (error) {
console.error('Error during cleanup:', error);
throw error;
}
}
/**
* Sanitize file name for storage
* @param {string} fileName - Original file name
* @returns {string} Sanitized file name
*/
sanitizeFileName(fileName) {
return fileName.replace(/[^a-zA-Z0-9._-]/g, '_');
}
}
module.exports = EnhancedDiffProcessingService;

509
services/git-integration/src/services/enhanced-webhook.service.js Normal file
View File

@ -0,0 +1,509 @@
const { v4: uuidv4 } = require('uuid');
const database = require('../config/database');
const CommitTrackingService = require('./commit-tracking.service');
class EnhancedWebhookService {
constructor() {
this.db = database;
this.commitTrackingService = new CommitTrackingService();
}
/**
* Process GitHub webhook
* @param {Object} payload - GitHub webhook payload
* @param {Object} headers - Request headers
* @returns {Object} Processing result
*/
async processGitHubWebhook(payload, headers) {
try {
const deliveryId = headers['x-github-delivery'];
const eventType = headers['x-github-event'];
// Store webhook record
const webhookRecord = await this.storeGitHubWebhook({
delivery_id: deliveryId,
event_type: eventType,
action: payload.action,
owner_name: payload.repository?.owner?.login,
repository_name: payload.repository?.name,
ref: payload.ref,
before_sha: payload.before,
after_sha: payload.after,
commit_count: payload.commits?.length || 0,
payload: payload
});
// Process based on event type
let processingResult = null;
if (eventType === 'push') {
processingResult = await this.processPushEvent(payload, 'github');
} else if (eventType === 'pull_request') {
processingResult = await this.processPullRequestEvent(payload, 'github');
}
// Update webhook as processed
await this.markWebhookProcessed(webhookRecord.id, 'github');
return {
success: true,
webhook_id: webhookRecord.id,
event_type: eventType,
processing_result: processingResult
};
} catch (error) {
console.error('Error processing GitHub webhook:', error);
throw error;
}
}
/**
* Process GitLab webhook
* @param {Object} payload - GitLab webhook payload
* @param {Object} headers - Request headers
* @returns {Object} Processing result
*/
async processGitLabWebhook(payload, headers) {
try {
const deliveryId = headers['x-gitlab-event-uuid'];
const eventType = headers['x-gitlab-event'];
// Store webhook record
const webhookRecord = await this.storeGitLabWebhook({
delivery_id: deliveryId,
event_type: eventType,
action: payload.object_kind,
owner_name: payload.project?.namespace || payload.user_username,
repository_name: payload.project?.name || payload.repository?.name,
ref: payload.ref,
before_sha: payload.before,
after_sha: payload.after,
commit_count: payload.total_commits_count || payload.commits?.length || 0,
payload: payload
});
// Process based on event type
let processingResult = null;
if (eventType === 'Push Hook' || payload.object_kind === 'push') {
processingResult = await this.processPushEvent(payload, 'gitlab');
} else if (eventType === 'Merge Request Hook' || payload.object_kind === 'merge_request') {
processingResult = await this.processPullRequestEvent(payload, 'gitlab');
}
return {
success: true,
webhook_id: webhookRecord.id,
event_type: eventType,
processing_result: processingResult
};
} catch (error) {
console.error('Error processing GitLab webhook:', error);
throw error;
}
}
/**
* Process Bitbucket webhook
* @param {Object} payload - Bitbucket webhook payload
* @param {Object} headers - Request headers
* @returns {Object} Processing result
*/
async processBitbucketWebhook(payload, headers) {
try {
const deliveryId = headers['x-request-uuid'];
const eventType = headers['x-event-key'];
// Store webhook record
const webhookRecord = await this.storeBitbucketWebhook({
delivery_id: deliveryId,
event_type: eventType,
action: payload.action || 'push',
owner_name: payload.repository?.owner?.username || payload.repository?.workspace?.slug,
repository_name: payload.repository?.name,
ref: payload.push?.changes?.[0]?.new?.name,
before_sha: payload.push?.changes?.[0]?.old?.target?.hash,
after_sha: payload.push?.changes?.[0]?.new?.target?.hash,
commit_count: payload.push?.changes?.[0]?.commits?.length || 0,
payload: payload
});
// Process based on event type
let processingResult = null;
if (eventType === 'repo:push') {
processingResult = await this.processPushEvent(payload, 'bitbucket');
} else if (eventType === 'pullrequest:created' || eventType === 'pullrequest:updated') {
processingResult = await this.processPullRequestEvent(payload, 'bitbucket');
}
return {
success: true,
webhook_id: webhookRecord.id,
event_type: eventType,
processing_result: processingResult
};
} catch (error) {
console.error('Error processing Bitbucket webhook:', error);
throw error;
}
}
/**
* Process Gitea webhook
* @param {Object} payload - Gitea webhook payload
* @param {Object} headers - Request headers
* @returns {Object} Processing result
*/
async processGiteaWebhook(payload, headers) {
try {
const deliveryId = headers['x-gitea-delivery'];
const eventType = headers['x-gitea-event'];
// Store webhook record
const webhookRecord = await this.storeGiteaWebhook({
delivery_id: deliveryId,
event_type: eventType,
action: payload.action,
owner_name: payload.repository?.owner?.login,
repository_name: payload.repository?.name,
ref: payload.ref,
before_sha: payload.before,
after_sha: payload.after,
commit_count: payload.commits?.length || 0,
payload: payload
});
// Process based on event type
let processingResult = null;
if (eventType === 'push') {
processingResult = await this.processPushEvent(payload, 'gitea');
} else if (eventType === 'pull_request') {
processingResult = await this.processPullRequestEvent(payload, 'gitea');
}
return {
success: true,
webhook_id: webhookRecord.id,
event_type: eventType,
processing_result: processingResult
};
} catch (error) {
console.error('Error processing Gitea webhook:', error);
throw error;
}
}
/**
* Process push event from any provider
* @param {Object} payload - Webhook payload
* @param {string} provider - Git provider (github, gitlab, bitbucket, gitea)
* @returns {Object} Processing result
*/
async processPushEvent(payload, provider) {
try {
// Find repository record
const repositoryId = await this.findRepositoryId(payload, provider);
if (!repositoryId) {
console.warn(`Repository not found for ${provider} push event`);
return { processed_commits: 0, message: 'Repository not found' };
}
// Extract commits based on provider format
const commits = this.extractCommitsFromPayload(payload, provider);
if (!commits || commits.length === 0) {
return { processed_commits: 0, message: 'No commits in payload' };
}
// Process commits using commit tracking service
const processedCommits = await this.commitTrackingService.processWebhookCommits(
{ commits },
repositoryId
);
// Update repository last sync info
await this.updateRepositorySync(repositoryId, payload, provider);
return {
processed_commits: processedCommits.length,
commits: processedCommits,
repository_id: repositoryId
};
} catch (error) {
console.error('Error processing push event:', error);
throw error;
}
}
/**
* Process pull request event from any provider
* @param {Object} payload - Webhook payload
* @param {string} provider - Git provider
* @returns {Object} Processing result
*/
async processPullRequestEvent(payload, provider) {
try {
// For now, just log PR events - can be extended later
console.log(`Processing ${provider} pull request event:`, {
action: payload.action,
pr_number: payload.number || payload.pull_request?.number || payload.pullrequest?.id,
title: payload.pull_request?.title || payload.pullrequest?.title
});
return {
event_type: 'pull_request',
action: payload.action,
message: 'Pull request event logged'
};
} catch (error) {
console.error('Error processing pull request event:', error);
throw error;
}
}
/**
* Extract commits from webhook payload based on provider
* @param {Object} payload - Webhook payload
* @param {string} provider - Git provider
* @returns {Array} Normalized commits array
*/
extractCommitsFromPayload(payload, provider) {
switch (provider) {
case 'github':
case 'gitea':
return payload.commits || [];
case 'gitlab':
return payload.commits || [];
case 'bitbucket':
return payload.push?.changes?.[0]?.commits || [];
default:
return [];
}
}
/**
* Find repository ID based on webhook payload
* @param {Object} payload - Webhook payload
* @param {string} provider - Git provider
* @returns {string|null} Repository ID
*/
async findRepositoryId(payload, provider) {
try {
let ownerName, repositoryName;
switch (provider) {
case 'github':
case 'gitea':
ownerName = payload.repository?.owner?.login;
repositoryName = payload.repository?.name;
break;
case 'gitlab':
ownerName = payload.project?.namespace || payload.user_username;
repositoryName = payload.project?.name || payload.repository?.name;
break;
case 'bitbucket':
ownerName = payload.repository?.owner?.username || payload.repository?.workspace?.slug;
repositoryName = payload.repository?.name;
break;
default:
return null;
}
if (!ownerName || !repositoryName) {
return null;
}
const query = `
SELECT id FROM github_repositories
WHERE owner_name = $1 AND repository_name = $2
LIMIT 1
`;
const result = await this.db.query(query, [ownerName, repositoryName]);
return result.rows[0]?.id || null;
} catch (error) {
console.error('Error finding repository ID:', error);
return null;
}
}
/**
* Update repository sync information
* @param {string} repositoryId - Repository ID
* @param {Object} payload - Webhook payload
* @param {string} provider - Git provider
*/
async updateRepositorySync(repositoryId, payload, provider) {
try {
let afterSha;
switch (provider) {
case 'github':
case 'gitea':
afterSha = payload.after;
break;
case 'gitlab':
afterSha = payload.after;
break;
case 'bitbucket':
afterSha = payload.push?.changes?.[0]?.new?.target?.hash;
break;
}
if (afterSha) {
const query = `
UPDATE github_repositories
SET last_synced_at = NOW(),
last_synced_commit_sha = $2,
sync_status = 'completed'
WHERE id = $1
`;
await this.db.query(query, [repositoryId, afterSha]);
}
} catch (error) {
console.error('Error updating repository sync:', error);
// Don't throw - this is not critical
}
}
// Webhook storage methods
async storeGitHubWebhook(webhookData) {
const query = `
INSERT INTO github_webhooks (
id, delivery_id, event_type, action, owner_name, repository_name,
ref, before_sha, after_sha, commit_count, payload, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, NOW(), NOW())
RETURNING *
`;
const values = [
uuidv4(),
webhookData.delivery_id,
webhookData.event_type,
webhookData.action,
webhookData.owner_name,
webhookData.repository_name,
webhookData.ref,
webhookData.before_sha,
webhookData.after_sha,
webhookData.commit_count,
JSON.stringify(webhookData.payload)
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async storeGitLabWebhook(webhookData) {
const query = `
INSERT INTO gitlab_webhooks (
id, delivery_id, event_type, action, owner_name, repository_name,
ref, before_sha, after_sha, commit_count, payload, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, NOW(), NOW())
RETURNING *
`;
const values = [
uuidv4(),
webhookData.delivery_id,
webhookData.event_type,
webhookData.action,
webhookData.owner_name,
webhookData.repository_name,
webhookData.ref,
webhookData.before_sha,
webhookData.after_sha,
webhookData.commit_count,
JSON.stringify(webhookData.payload)
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async storeBitbucketWebhook(webhookData) {
const query = `
INSERT INTO bitbucket_webhooks (
id, delivery_id, event_type, action, owner_name, repository_name,
ref, before_sha, after_sha, commit_count, payload, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, NOW(), NOW())
RETURNING *
`;
const values = [
uuidv4(),
webhookData.delivery_id,
webhookData.event_type,
webhookData.action,
webhookData.owner_name,
webhookData.repository_name,
webhookData.ref,
webhookData.before_sha,
webhookData.after_sha,
webhookData.commit_count,
JSON.stringify(webhookData.payload)
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async storeGiteaWebhook(webhookData) {
const query = `
INSERT INTO gitea_webhooks (
id, delivery_id, event_type, action, owner_name, repository_name,
ref, before_sha, after_sha, commit_count, payload, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, NOW(), NOW())
RETURNING *
`;
const values = [
uuidv4(),
webhookData.delivery_id,
webhookData.event_type,
webhookData.action,
webhookData.owner_name,
webhookData.repository_name,
webhookData.ref,
webhookData.before_sha,
webhookData.after_sha,
webhookData.commit_count,
JSON.stringify(webhookData.payload)
];
const result = await this.db.query(query, values);
return result.rows[0];
}
async markWebhookProcessed(webhookId, provider) {
const tableMap = {
'github': 'github_webhooks',
'gitlab': 'gitlab_webhooks',
'bitbucket': 'bitbucket_webhooks',
'gitea': 'gitea_webhooks'
};
const tableName = tableMap[provider];
if (!tableName) return;
const query = `
UPDATE ${tableName}
SET processed_at = NOW(), updated_at = NOW()
WHERE id = $1
`;
await this.db.query(query, [webhookId]);
}
}
module.exports = EnhancedWebhookService;

54
services/git-integration/src/services/file-storage.service.js
View File

@ -178,10 +178,7 @@ class FileStorageService {
const fileRecord = fileResult.rows[0];
// Process file content if it's a text file and not too large (< 10MB)
if (!isBinary && stats.size < 10 * 1024 * 1024) {
await this.processFileContent(fileRecord.id, absolutePath, extension);
}
return fileRecord;
@ -191,36 +188,7 @@ class FileStorageService {
}
}
// Process and store file content
async processFileContent(fileId, absolutePath, extension) {
try {
const content = fs.readFileSync(absolutePath, 'utf-8');
const lines = content.split('\n');
const preview = content.substring(0, 1000);
const language = this.languageMap[extension] || 'text';
const contentQuery = `
INSERT INTO repository_file_contents (
file_id, content_text, content_preview, language_detected,
line_count, char_count
) VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (file_id) DO UPDATE SET
content_text = $2,
content_preview = $3,
language_detected = $4,
line_count = $5,
char_count = $6,
updated_at = NOW()
`;
await database.query(contentQuery, [
fileId, content, preview, language, lines.length, content.length
]);
} catch (error) {
console.warn(`Error processing file content for file ID ${fileId}:`, error.message);
}
}
// Complete storage process for a repository
async completeRepositoryStorage(storageId) {
@ -307,10 +275,9 @@ class FileStorageService {
// Get files in a directory
async getDirectoryFiles(repositoryId, directoryPath = '') {
const query = `
SELECT rf.*, rfc.language_detected, rfc.line_count
SELECT rf.*
FROM repository_files rf
LEFT JOIN repository_directories rd ON rf.directory_id = rd.id
LEFT JOIN repository_file_contents rfc ON rf.id = rfc.file_id
WHERE rf.repository_id = $1 AND rd.relative_path = $2
ORDER BY rf.filename
`;
@ -321,19 +288,7 @@ class FileStorageService {
// Search files by content
async searchFileContent(repositoryId, searchQuery) {
const query = `
SELECT rf.filename, rf.relative_path, rfc.language_detected,
ts_rank_cd(to_tsvector('english', rfc.content_text), plainto_tsquery('english', $2)) as rank
FROM repository_files rf
JOIN repository_file_contents rfc ON rf.id = rfc.file_id
WHERE rf.repository_id = $1
AND to_tsvector('english', rfc.content_text) @@ plainto_tsquery('english', $2)
ORDER BY rank DESC, rf.filename
LIMIT 50
`;
const result = await database.query(query, [repositoryId, searchQuery]);
return result.rows;
return [];
}
// Utility methods
@ -373,7 +328,6 @@ class FileStorageService {
// Clean up storage for a repository
async cleanupRepositoryStorage(repositoryId) {
const queries = [
'DELETE FROM repository_file_contents WHERE file_id IN (SELECT id FROM repository_files WHERE repository_id = $1)',
'DELETE FROM repository_files WHERE repository_id = $1',
'DELETE FROM repository_directories WHERE repository_id = $1',
'DELETE FROM repository_storage WHERE repository_id = $1'

20
services/git-integration/src/services/git-repo.service.js
View File

@ -120,6 +120,20 @@ class GitRepoService {
}
async getDiff(repoPath, fromSha, toSha, options = { patch: true }) {
// Ensure both SHAs exist locally; if using shallow clone, fetch missing objects
try {
if (fromSha) {
await this.runGit(repoPath, ['cat-file', '-e', `${fromSha}^{commit}`]).catch(async () => {
await this.runGit(repoPath, ['fetch', '--depth', '200', 'origin', fromSha]);
});
}
if (toSha && toSha !== 'HEAD') {
await this.runGit(repoPath, ['cat-file', '-e', `${toSha}^{commit}`]).catch(async () => {
await this.runGit(repoPath, ['fetch', '--depth', '200', 'origin', toSha]);
});
}
} catch (_) {}
const range = fromSha && toSha ? `${fromSha}..${toSha}` : toSha ? `${toSha}^..${toSha}` : '';
const mode = options.patch ? '--patch' : '--name-status';
const args = ['diff', mode];
@ -129,6 +143,12 @@ class GitRepoService {
}
async getChangedFilesSince(repoPath, sinceSha) {
// Ensure SHA exists locally in case of shallow clone
try {
await this.runGit(repoPath, ['cat-file', '-e', `${sinceSha}^{commit}`]).catch(async () => {
await this.runGit(repoPath, ['fetch', '--depth', '200', 'origin', sinceSha]);
});
} catch (_) {}
const output = await this.runGit(repoPath, ['diff', '--name-status', `${sinceSha}..HEAD`]);
const lines = output.split('\n').filter(Boolean);
return lines.map(line => {

139
services/git-integration/src/services/gitea-oauth.js
View File

@ -1,6 +1,8 @@
// services/gitea-oauth.js
const database = require('../config/database');
const axios = require('axios');
class GiteaOAuthService {
constructor() {
this.clientId = process.env.GITEA_CLIENT_ID;
@ -17,42 +19,135 @@ class GiteaOAuthService {
redirect_uri: this.redirectUri,
response_type: 'code',
// Request both user and repository read scopes
scope: 'read:user read:repository',
scope: 'read:user read:repository write:repository',
state
});
return `${authUrl}?${params.toString()}`;
const fullUrl = `${authUrl}?${params.toString()}`;
console.log(`🔗 [GITEA OAUTH] Generated auth URL: ${fullUrl}`);
return fullUrl;
}
async exchangeCodeForToken(code) {
const tokenUrl = `${this.baseUrl}/login/oauth/access_token`;
const resp = await fetch(tokenUrl, {
method: 'POST',
headers: { 'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded' },
body: new URLSearchParams({
console.log(`🔄 [GITEA OAUTH] Exchanging code for token at: ${tokenUrl}`);
console.log(`🔧 [GITEA OAUTH] Config - Base URL: ${this.baseUrl}, Client ID: ${this.clientId?.substring(0, 8)}...`);
// Validate required configuration
if (!this.clientId) {
throw new Error('GITEA_CLIENT_ID is not configured');
}
if (!this.clientSecret) {
throw new Error('GITEA_CLIENT_SECRET is not configured');
}
try {
const response = await axios.post(tokenUrl, new URLSearchParams({
client_id: this.clientId,
client_secret: this.clientSecret,
code,
grant_type: 'authorization_code',
redirect_uri: this.redirectUri
})
});
let data = null;
try { data = await resp.json(); } catch (_) { data = null; }
if (!resp.ok || data?.error) {
const detail = data?.error_description || data?.error || (await resp.text().catch(() => '')) || 'unknown_error';
throw new Error(`Gitea token exchange failed: ${detail}`);
redirect_uri: this.redirectUri,
scope: 'read:user read:repository write:repository'
}), {
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Accept': 'application/json',
'User-Agent': 'CodeNuk-GitIntegration/1.0'
},
timeout: 30000,
maxRedirects: 0,
// Add network configuration to handle connectivity issues
httpsAgent: new (require('https').Agent)({
keepAlive: true,
timeout: 30000,
// Force IPv4 to avoid IPv6 connectivity issues
family: 4
}),
// Add retry configuration
validateStatus: function (status) {
return status >= 200 && status < 300;
}
});
console.log(`📥 [GITEA OAUTH] Response status: ${response.status} ${response.statusText}`);
console.log(`📥 [GITEA OAUTH] Response data:`, response.data);
if (response.data.error) {
console.error(`❌ [GITEA OAUTH] Token exchange failed:`, response.data);
throw new Error(response.data.error_description || response.data.error || 'Gitea token exchange failed');
}
if (!response.data.access_token) {
console.error(`❌ [GITEA OAUTH] No access token in response:`, response.data);
throw new Error('No access token received from Gitea OAuth');
}
console.log(`✅ [GITEA OAUTH] Token exchange successful`);
return response.data.access_token;
} catch (e) {
console.error(`❌ [GITEA OAUTH] Token exchange error:`, e);
// Handle AggregateError (multiple network errors)
if (e.name === 'AggregateError' && e.errors && e.errors.length > 0) {
const firstError = e.errors[0];
if (firstError.code === 'ETIMEDOUT') {
throw new Error(`Gitea OAuth timeout: Request to ${tokenUrl} timed out after 30 seconds`);
} else if (firstError.code === 'ENOTFOUND' || firstError.code === 'ECONNREFUSED') {
throw new Error(`Gitea OAuth network error: Cannot connect to ${this.baseUrl}. Please check your network connection and GITEA_BASE_URL configuration`);
} else {
throw new Error(`Gitea OAuth network error: ${firstError.message || 'Connection failed'}`);
}
}
if (e.code === 'ECONNABORTED' || e.message.includes('timeout')) {
throw new Error(`Gitea OAuth timeout: Request to ${tokenUrl} timed out after 30 seconds`);
} else if (e.code === 'ENOTFOUND' || e.code === 'ECONNREFUSED' || e.message.includes('Network Error')) {
throw new Error(`Gitea OAuth network error: Cannot connect to ${this.baseUrl}. Please check your network connection and GITEA_BASE_URL configuration`);
} else if (e.response) {
// Handle HTTP error responses
const status = e.response.status;
const data = e.response.data;
throw new Error(`Gitea OAuth HTTP error ${status}: ${JSON.stringify(data)}`);
} else {
throw new Error(`Gitea OAuth error: ${e.message || 'Unknown error occurred'}`);
}
}
return data.access_token;
}
async getUserInfo(accessToken) {
const resp = await fetch(`${this.baseUrl}/api/v1/user`, { headers: { Authorization: `Bearer ${accessToken}` } });
if (!resp.ok) {
let txt = '';
try { txt = await resp.text(); } catch (_) {}
throw new Error(`Failed to fetch Gitea user (status ${resp.status}): ${txt}`);
const userUrl = `${this.baseUrl}/api/v1/user`;
console.log(`🔄 [GITEA OAUTH] Fetching user info from: ${userUrl}`);
try {
const response = await axios.get(userUrl, {
headers: {
'Authorization': `token ${accessToken}`,
'Accept': 'application/json',
'User-Agent': 'CodeNuk-GitIntegration/1.0'
},
timeout: 15000,
// Add network configuration to handle connectivity issues
httpsAgent: new (require('https').Agent)({
keepAlive: true,
timeout: 15000,
// Force IPv4 to avoid IPv6 connectivity issues
family: 4
}),
// Add retry configuration
validateStatus: function (status) {
return status >= 200 && status < 300;
}
});
console.log(`📥 [GITEA OAUTH] User info response status: ${response.status} ${response.statusText}`);
console.log(`✅ [GITEA OAUTH] User info retrieved successfully for: ${response.data.login || response.data.username}`);
return response.data;
} catch (e) {
console.error(`❌ [GITEA OAUTH] User info error:`, e);
if (e.response) {
console.error(`❌ [GITEA OAUTH] User info failed:`, e.response.data);
throw new Error(`Failed to fetch Gitea user (${e.response.status}): ${JSON.stringify(e.response.data)}`);
} else if (e.code === 'ECONNABORTED' || e.message.includes('timeout')) {
throw new Error('Gitea user info timeout: Request timed out after 15 seconds');
} else if (e.code === 'ENOTFOUND' || e.code === 'ECONNREFUSED' || e.message.includes('Network Error')) {
throw new Error(`Gitea user info network error: Cannot connect to ${this.baseUrl}`);
} else {
throw e;
}
}
return await resp.json();
}
async storeToken(accessToken, user) {
@ -61,7 +156,7 @@ class GiteaOAuthService {
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (id) DO UPDATE SET access_token = EXCLUDED.access_token, gitea_username = EXCLUDED.gitea_username, gitea_user_id = EXCLUDED.gitea_user_id, scopes = EXCLUDED.scopes, expires_at = EXCLUDED.expires_at, updated_at = NOW()
RETURNING *`,
[accessToken, user.login, user.id, JSON.stringify(['read:repository']), null]
[accessToken, user.login, user.id, JSON.stringify(['read:user','read:repository']), null]
);
return result.rows[0];
}

60
services/git-integration/src/services/providers/bitbucket.adapter.js
View File

@ -87,14 +87,28 @@ class BitbucketAdapter extends VcsProviderInterface {
if (!callbackUrl) return { created: false, reason: 'missing_callback_url' };
const token = await this.oauth.getToken();
if (!token?.access_token) return { created: false, reason: 'missing_token' };
// Bitbucket Cloud webhooks don't support shared secret directly; create basic push webhook
const resp = await fetch(`https://api.bitbucket.org/2.0/repositories/${owner}/${repo}/hooks`, {
// Bitbucket Cloud requires repository:admin and webhook scopes
const hooksUrl = `https://api.bitbucket.org/2.0/repositories/${owner}/${repo}/hooks`;
// Avoid duplicates: list existing hooks first
try {
const listResp = await fetch(hooksUrl, { headers: { Authorization: `Bearer ${token.access_token}` } });
if (listResp.ok) {
const data = await listResp.json();
const existing = (data.values || []).find(h => h.url === callbackUrl);
if (existing) {
return { created: false, reason: 'already_exists', hook_id: existing.uuid || existing.id };
}
}
} catch (_) {}
// Create push webhook
const resp = await fetch(hooksUrl, {
method: 'POST',
headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${token.access_token}` },
body: JSON.stringify({ description: 'CodeNuk Git Integration', url: callbackUrl, active: true, events: ['repo:push'] })
});
if (resp.ok) { const d = await resp.json(); return { created: true, hook_id: d.uuid || d.id }; }
return { created: false, reason: `status_${resp.status}` };
const detail = await resp.text().catch(() => '');
return { created: false, reason: `status_${resp.status}`, detail };
} catch (e) {
return { created: false, error: e.message };
}
@ -111,13 +125,28 @@ class BitbucketAdapter extends VcsProviderInterface {
} else {
repoPath = await this.gitRepoService.cloneIfMissingWithHost(owner, repo, branch, this.host);
}
// Fetch and fast-forward with auth header for private repos
let beforeSha = await this.gitRepoService.getHeadSha(repoPath);
let afterSha = beforeSha;
try {
if (token?.access_token) {
// Use extraheader for both fetch and pull
await this.gitRepoService.runGit(repoPath, ['-c', `http.extraheader=Authorization: Bearer ${token.access_token}`, 'fetch', '--all', '--prune']);
await this.gitRepoService.runGit(repoPath, ['checkout', branch]);
await this.gitRepoService.runGit(repoPath, ['-c', `http.extraheader=Authorization: Bearer ${token.access_token}`, 'pull', '--ff-only', 'origin', branch]);
} else {
await this.gitRepoService.fetchAndFastForward(repoPath, branch);
}
afterSha = await this.gitRepoService.getHeadSha(repoPath);
} catch (_) {}
storageRecord = await this.fileStorageService.initializeRepositoryStorage(repositoryId, repoPath);
await this.fileStorageService.processDirectoryStructure(storageRecord.id, repositoryId, repoPath);
const finalStorage = await this.fileStorageService.completeRepositoryStorage(storageRecord.id);
// Get the current HEAD commit SHA and update the repository record
try {
const headSha = await this.gitRepoService.getHeadSha(repoPath);
const headSha = afterSha || (await this.gitRepoService.getHeadSha(repoPath));
await database.query(
'UPDATE github_repositories SET last_synced_at = NOW(), last_synced_commit_sha = $1, updated_at = NOW() WHERE id = $2',
[headSha, repositoryId]
@ -129,7 +158,7 @@ class BitbucketAdapter extends VcsProviderInterface {
[repositoryId]
);
}
return { success: true, method: 'git', targetDir: repoPath, storage: finalStorage };
return { success: true, method: 'git', targetDir: repoPath, beforeSha, afterSha, storage: finalStorage };
} catch (e) {
if (storageRecord) await this.fileStorageService.markStorageFailed(storageRecord.id, e.message);
return { success: false, error: e.message };
@ -148,11 +177,32 @@ class BitbucketAdapter extends VcsProviderInterface {
async getRepositoryDiff(owner, repo, branch, fromSha, toSha) {
const repoPath = this.gitRepoService.getLocalRepoPath(owner, repo, branch);
// Fast-forward before diff; include auth for private repos
try {
const token = await this.oauth.getToken();
if (token?.access_token) {
await this.gitRepoService.runGit(repoPath, ['-c', `http.extraheader=Authorization: Bearer ${token.access_token}`, 'fetch', '--all', '--prune']);
await this.gitRepoService.runGit(repoPath, ['checkout', branch]);
await this.gitRepoService.runGit(repoPath, ['-c', `http.extraheader=Authorization: Bearer ${token.access_token}`, 'pull', '--ff-only', 'origin', branch]);
} else {
await this.gitRepoService.fetchAndFastForward(repoPath, branch);
}
} catch (_) {}
return await this.gitRepoService.getDiff(repoPath, fromSha || null, toSha || 'HEAD', { patch: true });
}
async getRepositoryChangesSince(owner, repo, branch, sinceSha) {
const repoPath = this.gitRepoService.getLocalRepoPath(owner, repo, branch);
try {
const token = await this.oauth.getToken();
if (token?.access_token) {
await this.gitRepoService.runGit(repoPath, ['-c', `http.extraheader=Authorization: Bearer ${token.access_token}`, 'fetch', '--all', '--prune']);
await this.gitRepoService.runGit(repoPath, ['checkout', branch]);
await this.gitRepoService.runGit(repoPath, ['-c', `http.extraheader=Authorization: Bearer ${token.access_token}`, 'pull', '--ff-only', 'origin', branch]);
} else {
await this.gitRepoService.fetchAndFastForward(repoPath, branch);
}
} catch (_) {}
return await this.gitRepoService.getChangedFilesSince(repoPath, sinceSha);
}

205
services/git-integration/src/services/providers/gitea.adapter.js
View File

@ -3,6 +3,8 @@ const VcsProviderInterface = require('../vcs-provider.interface');
const FileStorageService = require('../file-storage.service');
const GitRepoService = require('../git-repo.service');
const GiteaOAuthService = require('../gitea-oauth');
const axios = require('axios');
const https = require('https');
class GiteaAdapter extends VcsProviderInterface {
constructor() {
@ -33,32 +35,89 @@ class GiteaAdapter extends VcsProviderInterface {
const token = await this.oauth.getToken();
const base = (process.env.GITEA_BASE_URL || 'https://gitea.com').replace(/\/$/, '');
console.log(`🔍 [GITEA] Checking repository access for: ${owner}/${repo}`);
console.log(`🔍 [GITEA] Token available: ${!!token?.access_token}`);
console.log(`🔍 [GITEA] API base URL: ${base}`);
try {
// Always try with authentication first (like GitHub behavior)
if (token?.access_token) {
const resp = await fetch(`${base}/api/v1/repos/${owner}/${repo}`, { headers: { Authorization: `Bearer ${token.access_token}` } });
if (resp.status === 200) {
const d = await resp.json();
const url = `${base}/api/v1/repos/${owner}/${repo}`;
console.log(`🔍 [GITEA] Trying authenticated request to: ${url}`);
const response = await axios.get(url, {
headers: { Authorization: `token ${token.access_token}` },
httpsAgent: new https.Agent({
keepAlive: true,
timeout: 15000,
family: 4 // Force IPv4 to avoid IPv6 connectivity issues
}),
timeout: 15000,
validateStatus: function (status) {
return status >= 200 && status < 300; // Only consider 2xx as success
}
});
console.log(`🔍 [GITEA] Authenticated response status: ${response.status}`);
if (response.status === 200) {
const d = response.data;
const isPrivate = !!d.private;
console.log(`✅ [GITEA] Repository accessible via authentication, private: ${isPrivate}`);
return { exists: true, isPrivate, hasAccess: true, requiresAuth: isPrivate };
} else {
console.log(`❌ [GITEA] Authenticated request failed with status: ${response.status}`);
console.log(`❌ [GITEA] Error response: ${JSON.stringify(response.data)}`);
}
}
// No token or token failed: try without authentication
const resp = await fetch(`${base}/api/v1/repos/${owner}/${repo}`);
if (resp.status === 200) {
const d = await resp.json();
const url = `${base}/api/v1/repos/${owner}/${repo}`;
console.log(`🔍 [GITEA] Trying unauthenticated request to: ${url}`);
const response = await axios.get(url, {
httpsAgent: new https.Agent({
keepAlive: true,
timeout: 15000,
family: 4 // Force IPv4 to avoid IPv6 connectivity issues
}),
timeout: 15000,
validateStatus: function (status) {
return status >= 200 && status < 300; // Only consider 2xx as success
}
});
console.log(`🔍 [GITEA] Unauthenticated response status: ${response.status}`);
if (response.status === 200) {
const d = response.data;
console.log(`✅ [GITEA] Repository accessible without authentication, private: ${!!d.private}`);
return { exists: true, isPrivate: !!d.private, hasAccess: true, requiresAuth: false };
}
if (resp.status === 404 || resp.status === 403) {
// Repository exists but requires authentication (like GitHub behavior)
return { exists: resp.status !== 404 ? true : false, isPrivate: true, hasAccess: false, requiresAuth: true };
}
} catch (error) {
if (error.response) {
// The request was made and the server responded with a status code
// that falls out of the range of 2xx
console.log(`❌ [GITEA] Request failed with status: ${error.response.status}`);
console.log(`❌ [GITEA] Error response: ${JSON.stringify(error.response.data)}`);
if (error.response.status === 404 || error.response.status === 403) {
console.log(`🔍 [GITEA] Repository exists but requires authentication (status: ${error.response.status})`);
return { exists: error.response.status !== 404 ? true : false, isPrivate: true, hasAccess: false, requiresAuth: true };
}
} else if (error.request) {
// The request was made but no response was received
console.log(`❌ [GITEA] Network error: ${error.message}`);
} else {
// Something happened in setting up the request that triggered an Error
console.log(`❌ [GITEA] Exception occurred: ${error.message}`);
}
// If any error occurs, assume repository requires authentication
return { exists: false, isPrivate: null, hasAccess: false, requiresAuth: true, error: 'Repository not found or requires authentication' };
}
console.log(`❌ [GITEA] Falling through to default error case`);
return { exists: false, isPrivate: null, hasAccess: false, requiresAuth: true, error: 'Repository not found or requires authentication' };
}
@ -67,12 +126,22 @@ class GiteaAdapter extends VcsProviderInterface {
const base = (process.env.GITEA_BASE_URL || 'https://gitea.com').replace(/\/$/, '');
if (token?.access_token) {
try {
const resp = await fetch(`${base}/api/v1/repos/${owner}/${repo}`, { headers: { Authorization: `Bearer ${token.access_token}` } });
if (resp.ok) {
const d = await resp.json();
const response = await axios.get(`${base}/api/v1/repos/${owner}/${repo}`, {
headers: { Authorization: `token ${token.access_token}` },
httpsAgent: new https.Agent({
keepAlive: true,
timeout: 15000,
family: 4 // Force IPv4 to avoid IPv6 connectivity issues
}),
timeout: 15000
});
if (response.status === 200) {
const d = response.data;
return { full_name: d.full_name || `${owner}/${repo}`, visibility: d.private ? 'private' : 'public', default_branch: d.default_branch || 'main', updated_at: d.updated_at };
}
} catch (_) {}
} catch (error) {
console.log(`❌ [GITEA] Failed to fetch repository metadata: ${error.message}`);
}
}
return { full_name: `${owner}/${repo}`, visibility: 'public', default_branch: 'main', updated_at: new Date().toISOString() };
}
@ -83,20 +152,92 @@ class GiteaAdapter extends VcsProviderInterface {
async ensureRepositoryWebhook(owner, repo, callbackUrl) {
try {
if (!callbackUrl) return { created: false, reason: 'missing_callback_url' };
if (!callbackUrl) {
console.warn('⚠️ [GITEA] Webhook callbackUrl not provided; skipping webhook creation');
return { created: false, reason: 'missing_callback_url' };
}
const token = await this.oauth.getToken();
if (!token?.access_token) return { created: false, reason: 'missing_token' };
if (!token?.access_token) {
console.warn('⚠️ [GITEA] OAuth token not available; skipping webhook creation');
return { created: false, reason: 'missing_token' };
}
const base = (process.env.GITEA_BASE_URL || 'https://gitea.com').replace(/\/$/, '');
const secret = process.env.GITEA_WEBHOOK_SECRET || '';
const resp = await fetch(`${base}/api/v1/repos/${owner}/${repo}/hooks`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${token.access_token}` },
body: JSON.stringify({ type: 'gitea', config: { url: callbackUrl, content_type: 'json', secret: secret || undefined }, events: ['push'], active: true })
console.log(`🔗 [GITEA] Setting up webhook for ${owner}/${repo}`);
// First, list existing hooks to avoid duplicates
try {
const listResponse = await axios.get(`${base}/api/v1/repos/${owner}/${repo}/hooks`, {
headers: { Authorization: `token ${token.access_token}` },
httpsAgent: new https.Agent({
keepAlive: true,
timeout: 15000,
family: 4
}),
timeout: 15000
});
if (listResponse.status === 200) {
const existingHooks = listResponse.data;
// Check if a webhook with our callback URL already exists
const existingHook = existingHooks.find(hook =>
hook.config && hook.config.url === callbackUrl
);
if (existingHook) {
console.log(`✅ [GITEA] Webhook already exists (ID: ${existingHook.id})`);
return { created: false, reason: 'already_exists', hook_id: existingHook.id };
}
}
} catch (error) {
console.warn('⚠️ [GITEA] Could not list existing webhooks, continuing with creation attempt:', error.message);
}
// Create new webhook
const response = await axios.post(`${base}/api/v1/repos/${owner}/${repo}/hooks`, {
type: 'gitea',
config: {
url: callbackUrl,
content_type: 'json',
secret: secret || undefined
},
events: ['push'],
active: true
}, {
headers: {
'Content-Type': 'application/json',
Authorization: `token ${token.access_token}`
},
httpsAgent: new https.Agent({
keepAlive: true,
timeout: 15000,
family: 4
}),
timeout: 15000
});
if (resp.ok) { const d = await resp.json(); return { created: true, hook_id: d.id }; }
return { created: false, reason: `status_${resp.status}` };
} catch (e) {
return { created: false, error: e.message };
if (response.status === 200 || response.status === 201) {
const hookData = response.data;
console.log(`✅ [GITEA] Webhook created successfully (ID: ${hookData.id})`);
return { created: true, hook_id: hookData.id };
}
console.warn(`⚠️ [GITEA] Webhook creation failed with status: ${response.status}`);
return { created: false, reason: `status_${response.status}` };
} catch (error) {
// Common cases: insufficient permissions, private repo without correct scope
if (error.response) {
console.warn('⚠️ [GITEA] Webhook creation failed:', error.response.status, error.response.data?.message || error.message);
return { created: false, error: error.message, status: error.response.status };
} else {
console.warn('⚠️ [GITEA] Webhook creation failed:', error.message);
return { created: false, error: error.message };
}
}
}
@ -111,13 +252,22 @@ class GiteaAdapter extends VcsProviderInterface {
} else {
repoPath = await this.gitRepoService.cloneIfMissingWithHost(owner, repo, branch, this.host);
}
// Fetch and fast-forward to ensure latest commits are present
let beforeSha = await this.gitRepoService.getHeadSha(repoPath);
let afterSha = beforeSha;
try {
const res = await this.gitRepoService.fetchAndFastForward(repoPath, branch);
beforeSha = res.beforeSha || beforeSha;
afterSha = res.afterSha || afterSha;
} catch (_) {}
storageRecord = await this.fileStorageService.initializeRepositoryStorage(repositoryId, repoPath);
await this.fileStorageService.processDirectoryStructure(storageRecord.id, repositoryId, repoPath);
const finalStorage = await this.fileStorageService.completeRepositoryStorage(storageRecord.id);
// Get the current HEAD commit SHA and update the repository record
try {
const headSha = await this.gitRepoService.getHeadSha(repoPath);
const headSha = afterSha || (await this.gitRepoService.getHeadSha(repoPath));
await database.query(
'UPDATE github_repositories SET last_synced_at = NOW(), last_synced_commit_sha = $1, updated_at = NOW() WHERE id = $2',
[headSha, repositoryId]
@ -129,7 +279,7 @@ class GiteaAdapter extends VcsProviderInterface {
[repositoryId]
);
}
return { success: true, method: 'git', targetDir: repoPath, storage: finalStorage };
return { success: true, method: 'git', targetDir: repoPath, beforeSha, afterSha, storage: finalStorage };
} catch (e) {
if (storageRecord) await this.fileStorageService.markStorageFailed(storageRecord.id, e.message);
return { success: false, error: e.message };
@ -148,11 +298,14 @@ class GiteaAdapter extends VcsProviderInterface {
async getRepositoryDiff(owner, repo, branch, fromSha, toSha) {
const repoPath = this.gitRepoService.getLocalRepoPath(owner, repo, branch);
// Proactively fetch latest to ensure SHAs exist
try { await this.gitRepoService.fetchAndFastForward(repoPath, branch); } catch (_) {}
return await this.gitRepoService.getDiff(repoPath, fromSha || null, toSha || 'HEAD', { patch: true });
}
async getRepositoryChangesSince(owner, repo, branch, sinceSha) {
const repoPath = this.gitRepoService.getLocalRepoPath(owner, repo, branch);
try { await this.gitRepoService.fetchAndFastForward(repoPath, branch); } catch (_) {}
return await this.gitRepoService.getChangedFilesSince(repoPath, sinceSha);
}

121
services/git-integration/src/services/providers/gitlab.adapter.js
View File

@ -158,13 +158,126 @@ class GitlabAdapter extends VcsProviderInterface {
}
async getRepositoryDiff(owner, repo, branch, fromSha, toSha) {
const repoPath = this.gitRepoService.getLocalRepoPath(owner, repo, branch);
return await this.gitRepoService.getDiff(repoPath, fromSha || null, toSha || 'HEAD', { patch: true });
// Mirror robust GitHub behavior: ensure repo exists, handle main/master fallback,
// fetch required history for provided SHAs (handle shallow clones), then diff.
const preferredBranch = branch || 'main';
const alternateBranch = preferredBranch === 'main' ? 'master' : 'main';
const fs = require('fs');
const path = require('path');
let repoPath = this.gitRepoService.getLocalRepoPath(owner, repo, preferredBranch);
// Ensure repo exists locally (prefer OAuth)
try {
const token = await this.oauth.getToken().catch(() => null);
try {
if (token?.access_token) {
repoPath = await this.gitRepoService.cloneIfMissingWithAuth(owner, repo, preferredBranch, this.host, token.access_token, 'oauth2');
} else {
repoPath = await this.gitRepoService.cloneIfMissingWithHost(owner, repo, preferredBranch, this.host);
}
} catch (_) {
// Try alternate common default branch
try {
if (token?.access_token) {
repoPath = await this.gitRepoService.cloneIfMissingWithAuth(owner, repo, alternateBranch, this.host, token.access_token, 'oauth2');
} else {
repoPath = await this.gitRepoService.cloneIfMissingWithHost(owner, repo, alternateBranch, this.host);
}
} catch (_) {}
}
// If preferred path missing but alternate exists, use alternate
const altPath = this.gitRepoService.getLocalRepoPath(owner, repo, alternateBranch);
if ((!fs.existsSync(repoPath) || !fs.existsSync(path.join(repoPath, '.git'))) && fs.existsSync(altPath)) {
repoPath = altPath;
}
// Fetch and checkout; attempt preferred then alternate
try {
await this.gitRepoService.fetchAndFastForward(repoPath, preferredBranch);
} catch (_) {
try { await this.gitRepoService.fetchAndFastForward(repoPath, alternateBranch); } catch (_) {}
}
// Ensure both SHAs exist locally; if not, fetch them explicitly
const ensureShaPresent = async (sha) => {
if (!sha) return;
try {
await this.gitRepoService.runGit(repoPath, ['cat-file', '-e', `${sha}^{commit}`]);
} catch (_) {
// Try fetching just that object; if shallow, unshallow or fetch full history
try { await this.gitRepoService.runGit(repoPath, ['fetch', '--depth=2147483647', 'origin']); } catch (_) {}
try { await this.gitRepoService.runGit(repoPath, ['fetch', 'origin', sha]); } catch (_) {}
}
};
await ensureShaPresent(fromSha || null);
await ensureShaPresent(toSha || null);
return await this.gitRepoService.getDiff(repoPath, fromSha || null, toSha || 'HEAD', { patch: true });
} catch (error) {
const attempted = [
this.gitRepoService.getLocalRepoPath(owner, repo, preferredBranch),
this.gitRepoService.getLocalRepoPath(owner, repo, alternateBranch)
].join(' | ');
throw new Error(`${error.message} (attempted paths: ${attempted})`);
}
}
async getRepositoryChangesSince(owner, repo, branch, sinceSha) {
const repoPath = this.gitRepoService.getLocalRepoPath(owner, repo, branch);
return await this.gitRepoService.getChangedFilesSince(repoPath, sinceSha);
const preferredBranch = branch || 'main';
const alternateBranch = preferredBranch === 'main' ? 'master' : 'main';
const fs = require('fs');
const path = require('path');
let repoPath = this.gitRepoService.getLocalRepoPath(owner, repo, preferredBranch);
try {
const token = await this.oauth.getToken().catch(() => null);
try {
if (token?.access_token) {
repoPath = await this.gitRepoService.cloneIfMissingWithAuth(owner, repo, preferredBranch, this.host, token.access_token, 'oauth2');
} else {
repoPath = await this.gitRepoService.cloneIfMissingWithHost(owner, repo, preferredBranch, this.host);
}
} catch (_) {
try {
if (token?.access_token) {
repoPath = await this.gitRepoService.cloneIfMissingWithAuth(owner, repo, alternateBranch, this.host, token.access_token, 'oauth2');
} else {
repoPath = await this.gitRepoService.cloneIfMissingWithHost(owner, repo, alternateBranch, this.host);
}
} catch (_) {}
}
const altPath = this.gitRepoService.getLocalRepoPath(owner, repo, alternateBranch);
if ((!fs.existsSync(repoPath) || !fs.existsSync(path.join(repoPath, '.git'))) && fs.existsSync(altPath)) {
repoPath = altPath;
}
try {
await this.gitRepoService.fetchAndFastForward(repoPath, preferredBranch);
} catch (_) {
try { await this.gitRepoService.fetchAndFastForward(repoPath, alternateBranch); } catch (_) {}
}
// Ensure sinceSha exists locally
if (sinceSha) {
try {
await this.gitRepoService.runGit(repoPath, ['cat-file', '-e', `${sinceSha}^{commit}`]);
} catch (_) {
try { await this.gitRepoService.runGit(repoPath, ['fetch', '--depth=2147483647', 'origin']); } catch (_) {}
try { await this.gitRepoService.runGit(repoPath, ['fetch', 'origin', sinceSha]); } catch (_) {}
}
}
return await this.gitRepoService.getChangedFilesSince(repoPath, sinceSha);
} catch (error) {
const attempted = [
this.gitRepoService.getLocalRepoPath(owner, repo, preferredBranch),
this.gitRepoService.getLocalRepoPath(owner, repo, alternateBranch)
].join(' | ');
throw new Error(`${error.message} (attempted paths: ${attempted})`);
}
}
async cleanupRepositoryStorage(repositoryId) {

266
services/git-integration/src/services/vcs-webhook.service.js
View File

@ -1,11 +1,13 @@
// services/vcs-webhook.service.js
const database = require('../config/database');
const providerRegistry = require('./provider-registry');
const DiffProcessingService = require('./diff-processing.service');
class VcsWebhookService {
constructor() {
this._schemaChecked = false;
this._webhookEventColumns = new Map();
this.diffService = new DiffProcessingService();
}
// Process webhook events for any VCS provider
@ -38,7 +40,32 @@ class VcsWebhookService {
// Handle push events for any provider
async handlePushEvent(providerKey, payload) {
const { repository, project, ref, commits, pusher, user } = payload;
let { repository, project, ref, commits, pusher, user } = payload;
// Normalize Bitbucket push payload structure to the common fields
if (providerKey === 'bitbucket' && payload && payload.push && Array.isArray(payload.push.changes)) {
try {
const firstChange = payload.push.changes[0] || {};
const newRef = firstChange.new || {};
const oldRef = firstChange.old || {};
// Branch name
const branchName = newRef.name || oldRef.name || null;
// Compose a git-like ref
ref = branchName ? `refs/heads/${branchName}` : ref;
// Aggregate commits across changes
const allCommits = [];
for (const change of payload.push.changes) {
if (Array.isArray(change.commits)) {
allCommits.push(...change.commits);
}
}
commits = allCommits;
// Surface before/after hashes for persistence
payload.before = oldRef.target?.hash || payload.before || null;
payload.after = newRef.target?.hash || payload.after || null;
// Bitbucket sets repository at top-level; keep as-is if present
} catch (_) {}
}
// Build a provider-normalized repo object for extraction
let repoForExtraction = repository || {};
@ -85,12 +112,7 @@ class VcsWebhookService {
});
if (repoId) {
// Insert into repository_commit_events
await database.query(
`INSERT INTO repository_commit_events (repository_id, ref, before_sha, after_sha, commit_count)
VALUES ($1, $2, $3, $4, $5)`,
[repoId, ref, payload.before || null, payload.after || null, commitData.length]
);
// Persist per-commit details and file paths
if (commitData.length > 0) {
@ -114,23 +136,25 @@ class VcsWebhookService {
commit.url || null
]
);
const commitId = commitInsert.rows[0].id;
// Insert file changes
const addFiles = (paths = [], changeType) => paths.forEach(async (p) => {
try {
await database.query(
`INSERT INTO repository_commit_files (commit_id, change_type, file_path)
VALUES ($1, $2, $3)`,
[commitId, changeType, p]
);
} catch (_) {}
});
addFiles(commit.added || [], 'added');
addFiles(commit.modified || [], 'modified');
addFiles(commit.removed || [], 'removed');
// For Bitbucket, we'll skip file change insertion during webhook processing
// since the webhook doesn't include file changes. The background sync will
// handle this properly by fetching the changes from git directly.
if (providerKey !== 'bitbucket') {
// For other providers (GitHub, GitLab, Gitea), use the webhook data
const addFiles = (paths = [], changeType) => paths.forEach(async (p) => {
try {
await database.query(
`INSERT INTO repository_commit_files (commit_id, change_type, file_path)
VALUES ($1, $2, $3)`,
[commitId, changeType, p]
);
} catch (_) {}
});
addFiles(commit.added || [], 'added');
addFiles(commit.modified || [], 'modified');
addFiles(commit.removed || [], 'removed');
}
} catch (commitErr) {
console.warn('Failed to persist commit details:', commitErr.message);
}
@ -157,6 +181,10 @@ class VcsWebhookService {
repoId
);
// Process diffs for each commit after successful sync
if (downloadResult.success && downloadResult.targetDir) {
await this.processCommitDiffs(repoId, commitData, downloadResult.targetDir, providerKey);
}
await database.query(
'UPDATE github_repositories SET sync_status = $1, last_synced_at = NOW(), updated_at = NOW() WHERE id = $2',
[downloadResult.success ? 'synced' : 'error', repoId]
@ -250,9 +278,11 @@ class VcsWebhookService {
},
message: commit.message,
url: commit.links?.html?.href,
added: commit.added || [],
modified: commit.modified || [],
removed: commit.removed || []
// Bitbucket webhook doesn't include file changes in commit objects
// We'll fetch these from git directly during processing
added: [],
modified: [],
removed: []
}));
case 'gitea':
return commits.map(commit => ({
@ -361,94 +391,124 @@ class VcsWebhookService {
console.log(`Repository: ${payload.repository?.full_name || payload.repository?.path_with_namespace || 'Unknown'}`);
}
// Log webhook events for debugging and analytics
async logWebhookEvent(providerKey, eventType, action, repositoryFullName, metadata = {}, deliveryId = null, fullPayload = null) {
try {
await this._ensureWebhookEventsSchemaCached();
// Process diffs for commits in a push event (same as GitHub webhook service)
async processCommitDiffs(repositoryId, commits, repoLocalPath, providerKey = 'github') {
console.log(`🔄 Processing diffs for ${commits.length} commits in repository ${repositoryId}`);
// Build a flexible INSERT based on existing columns
const columns = [];
const placeholders = [];
const values = [];
let i = 1;
if (!Array.isArray(commits) || commits.length === 0) {
console.log('⚠️ No commits to process');
return;
}
columns.push('event_type');
placeholders.push(`$${i++}`);
values.push(eventType);
for (const commit of commits) {
try {
console.log(`📝 Processing diff for commit: ${commit.id}`);
const commitQuery = `
SELECT id FROM repository_commit_details
WHERE repository_id = $1 AND commit_sha = $2
`;
const commitResult = await database.query(commitQuery, [repositoryId, commit.id]);
if (commitResult.rows.length === 0) {
console.warn(`⚠️ Commit ${commit.id} not found in database, skipping diff processing`);
continue;
}
const commitId = commitResult.rows[0].id;
const parentSha = commit.parents && commit.parents.length > 0 ? commit.parents[0].id : null;
if (this._webhookEventColumns.has('action')) {
columns.push('action');
placeholders.push(`$${i++}`);
values.push(action || null);
}
if (this._webhookEventColumns.has('repository_full_name')) {
columns.push('repository_full_name');
placeholders.push(`$${i++}`);
values.push(repositoryFullName || null);
}
if (this._webhookEventColumns.has('delivery_id')) {
columns.push('delivery_id');
placeholders.push(`$${i++}`);
values.push(deliveryId || null);
}
if (this._webhookEventColumns.has('metadata')) {
columns.push('metadata');
placeholders.push(`$${i++}`);
values.push(JSON.stringify({ ...metadata, provider: providerKey }));
}
if (this._webhookEventColumns.has('event_payload')) {
columns.push('event_payload');
placeholders.push(`$${i++}`);
values.push(JSON.stringify(fullPayload || {}));
}
if (this._webhookEventColumns.has('received_at')) {
columns.push('received_at');
placeholders.push(`$${i++}`);
values.push(new Date());
}
if (this._webhookEventColumns.has('processing_status')) {
columns.push('processing_status');
placeholders.push(`$${i++}`);
values.push('pending');
}
// For Bitbucket, we need to ensure file changes are in the database first
if (providerKey === 'bitbucket') {
await this.ensureBitbucketFileChanges(commitId, commit, repoLocalPath);
}
const diffResult = await this.diffService.processCommitDiffs(
commitId,
repositoryId,
repoLocalPath,
parentSha,
commit.id
);
const query = `INSERT INTO webhook_events (${columns.join(', ')}) VALUES (${placeholders.join(', ')})`;
await database.query(query, values);
} catch (error) {
console.warn('Failed to log webhook event:', error.message);
if (diffResult.success) {
console.log(`✅ Successfully processed ${diffResult.processedFiles} file diffs for commit ${commit.id}`);
} else {
console.warn(`⚠️ Failed to process diffs for commit ${commit.id}: ${diffResult.error || diffResult.reason}`);
}
} catch (error) {
console.error(`❌ Error processing diff for commit ${commit.id}:`, error.message);
}
}
}
async _ensureWebhookEventsSchemaCached() {
if (this._schemaChecked) return;
// Ensure Bitbucket file changes are stored in the database
async ensureBitbucketFileChanges(commitId, commit, repoLocalPath) {
try {
const result = await database.query(
"SELECT column_name, is_nullable FROM information_schema.columns WHERE table_schema='public' AND table_name='webhook_events'"
);
for (const row of result.rows) {
this._webhookEventColumns.set(row.column_name, row.is_nullable);
}
} catch (e) {
console.warn('Could not introspect webhook_events schema:', e.message);
} finally {
this._schemaChecked = true;
}
}
// Get recent webhook events
async getRecentWebhookEvents(limit = 50) {
try {
const query = `
SELECT * FROM webhook_events
ORDER BY received_at DESC
LIMIT $1
// Check if file changes already exist for this commit
const existingFilesQuery = `
SELECT COUNT(*) as count FROM repository_commit_files WHERE commit_id = $1
`;
const existingResult = await database.query(existingFilesQuery, [commitId]);
const result = await database.query(query, [limit]);
return result.rows;
if (parseInt(existingResult.rows[0].count) > 0) {
console.log(`📁 File changes already exist for commit ${commit.id}`);
return;
}
// Get file changes from git
const GitRepoService = require('./git-repo.service');
const gitService = new GitRepoService();
const parentSha = commit.parents && commit.parents.length > 0 ? commit.parents[0].id : null;
const fromSha = parentSha || `${commit.id}~1`;
const fileChanges = await gitService.getChangedFilesSince(repoLocalPath, fromSha);
// Parse the git diff output and categorize files
const added = [];
const modified = [];
const removed = [];
for (const change of fileChanges) {
const { status, filePath } = change;
switch (status) {
case 'A':
added.push(filePath);
break;
case 'M':
modified.push(filePath);
break;
case 'D':
removed.push(filePath);
break;
case 'R':
// Renamed files - treat as modified for now
modified.push(filePath);
break;
default:
// Unknown status, treat as modified
modified.push(filePath);
}
}
// Insert file changes
const addFiles = (paths = [], changeType) => paths.forEach(async (p) => {
try {
await database.query(
`INSERT INTO repository_commit_files (commit_id, change_type, file_path)
VALUES ($1, $2, $3)`,
[commitId, changeType, p]
);
} catch (err) {
console.warn(`Failed to insert file change: ${err.message}`);
}
});
addFiles(added, 'added');
addFiles(modified, 'modified');
addFiles(removed, 'removed');
console.log(`📁 Inserted ${added.length + modified.length + removed.length} file changes for Bitbucket commit ${commit.id}`);
} catch (error) {
console.error('Failed to get webhook events:', error.message);
return [];
console.warn(`Failed to ensure Bitbucket file changes for commit ${commit.id}:`, error.message);
}
}
}

149
services/git-integration/src/services/webhook.service.js
View File

@ -2,6 +2,7 @@
const crypto = require('crypto');
const database = require('../config/database');
const GitHubIntegrationService = require('./github-integration.service');
const DiffProcessingService = require('./diff-processing.service');
class WebhookService {
constructor() {
@ -9,6 +10,7 @@ class WebhookService {
this._schemaChecked = false;
this._webhookEventColumns = new Map();
this.githubService = new GitHubIntegrationService();
this.diffService = new DiffProcessingService();
}
// Verify GitHub webhook signature
@ -97,11 +99,7 @@ class WebhookService {
);
if (repoId) {
await database.query(
`INSERT INTO repository_commit_events (repository_id, ref, before_sha, after_sha, commit_count)
VALUES ($1, $2, $3, $4, $5)`,
[repoId, ref, payload.before || null, payload.after || null, Array.isArray(commits) ? commits.length : 0]
);
// repository_commit_events table removed as requested
// Persist per-commit details and file paths (added/modified/removed)
if (Array.isArray(commits) && commits.length > 0) {
@ -165,6 +163,11 @@ class WebhookService {
repoId
);
// Process diffs for each commit after successful sync
if (downloadResult.success && downloadResult.targetDir) {
await this.processCommitDiffs(repoId, commits, downloadResult.targetDir);
}
await database.query(
'UPDATE github_repositories SET sync_status = $1, last_synced_at = NOW(), updated_at = NOW() WHERE id = $2',
[downloadResult.success ? 'synced' : 'error', repoId]
@ -265,97 +268,59 @@ class WebhookService {
console.log(`Zen: ${payload.zen || 'No zen message'}`);
}
// Log webhook events for debugging and analytics
async _ensureWebhookEventsSchemaCached() {
if (this._schemaChecked) return;
try {
const result = await database.query(
"SELECT column_name, is_nullable FROM information_schema.columns WHERE table_schema='public' AND table_name='webhook_events'"
);
for (const row of result.rows) {
this._webhookEventColumns.set(row.column_name, row.is_nullable);
// Process diffs for commits in a push event
async processCommitDiffs(repositoryId, commits, repoLocalPath) {
console.log(`🔄 Processing diffs for ${commits.length} commits in repository ${repositoryId}`);
if (!Array.isArray(commits) || commits.length === 0) {
console.log('⚠️ No commits to process');
return;
}
for (const commit of commits) {
try {
console.log(`📝 Processing diff for commit: ${commit.id}`);
// Get commit record from database
const commitQuery = `
SELECT id FROM repository_commit_details
WHERE repository_id = $1 AND commit_sha = $2
`;
const commitResult = await database.query(commitQuery, [repositoryId, commit.id]);
if (commitResult.rows.length === 0) {
console.warn(`⚠️ Commit ${commit.id} not found in database, skipping diff processing`);
continue;
}
const commitId = commitResult.rows[0].id;
// Get parent commit SHA for diff calculation
const parentSha = commit.parents && commit.parents.length > 0 ? commit.parents[0].id : null;
// Process the diff
const diffResult = await this.diffService.processCommitDiffs(
commitId,
repositoryId,
repoLocalPath,
parentSha,
commit.id
);
if (diffResult.success) {
console.log(`✅ Successfully processed ${diffResult.processedFiles} file diffs for commit ${commit.id}`);
} else {
console.warn(`⚠️ Failed to process diffs for commit ${commit.id}: ${diffResult.error || diffResult.reason}`);
}
} catch (error) {
console.error(`❌ Error processing diff for commit ${commit.id}:`, error.message);
}
} catch (e) {
// If schema check fails, proceed with best-effort insert
console.warn('Could not introspect webhook_events schema:', e.message);
} finally {
this._schemaChecked = true;
}
}
async logWebhookEvent(eventType, action, repositoryFullName, metadata = {}, deliveryId = null, fullPayload = null) {
try {
await this._ensureWebhookEventsSchemaCached();
// Build a flexible INSERT based on existing columns
const columns = [];
const placeholders = [];
const values = [];
let i = 1;
columns.push('event_type');
placeholders.push(`$${i++}`);
values.push(eventType);
if (this._webhookEventColumns.has('action')) {
columns.push('action');
placeholders.push(`$${i++}`);
values.push(action || null);
}
if (this._webhookEventColumns.has('repository_full_name')) {
columns.push('repository_full_name');
placeholders.push(`$${i++}`);
values.push(repositoryFullName || null);
}
if (this._webhookEventColumns.has('delivery_id')) {
columns.push('delivery_id');
placeholders.push(`$${i++}`);
values.push(deliveryId || null);
}
if (this._webhookEventColumns.has('metadata')) {
columns.push('metadata');
placeholders.push(`$${i++}`);
values.push(JSON.stringify(metadata || {}));
}
if (this._webhookEventColumns.has('event_payload')) {
columns.push('event_payload');
placeholders.push(`$${i++}`);
values.push(JSON.stringify(fullPayload || {}));
}
if (this._webhookEventColumns.has('received_at')) {
columns.push('received_at');
placeholders.push(`$${i++}`);
values.push(new Date());
}
if (this._webhookEventColumns.has('processing_status')) {
columns.push('processing_status');
placeholders.push(`$${i++}`);
values.push('pending');
}
const query = `INSERT INTO webhook_events (${columns.join(', ')}) VALUES (${placeholders.join(', ')})`;
await database.query(query, values);
} catch (error) {
console.warn('Failed to log webhook event:', error.message);
}
}
// Get recent webhook events
async getRecentWebhookEvents(limit = 50) {
try {
const query = `
SELECT * FROM webhook_events
ORDER BY received_at DESC
LIMIT $1
`;
const result = await database.query(query, [limit]);
return result.rows;
} catch (error) {
console.error('Failed to get webhook events:', error.message);
return [];
}
}
// webhook_events table removed as requested - logging functionality disabled
}
module.exports = WebhookService;

94
services/git-integration/test-repo.js
View File

@ -1,94 +0,0 @@
const GitHubIntegrationService = require('./src/services/github-integration.service');
const fs = require('fs');
const path = require('path');
async function testRepository() {
try {
console.log('🔍 Testing GitHub Repository Integration...');
const githubService = new GitHubIntegrationService();
const repositoryUrl = 'https://github.com/prakash6383206529/code-generator.git';
console.log(`📂 Testing repository: ${repositoryUrl}`);
// Parse the GitHub URL
const { owner, repo, branch } = githubService.parseGitHubUrl(repositoryUrl);
console.log(`📋 Parsed URL - Owner: ${owner}, Repo: ${repo}, Branch: ${branch || 'main'}`);
// Check if repository is public
try {
const unauthenticatedOctokit = new (require('@octokit/rest')).Octokit({
userAgent: 'CodeNuk-GitIntegration/1.0.0',
});
const { data: repoInfo } = await unauthenticatedOctokit.repos.get({ owner, repo });
const isPublic = !repoInfo.private;
console.log(`🔓 Repository is ${isPublic ? 'public' : 'private'}`);
console.log(`📊 Repository info:`);
console.log(` - Full Name: ${repoInfo.full_name}`);
console.log(` - Description: ${repoInfo.description || 'No description'}`);
console.log(` - Language: ${repoInfo.language || 'Unknown'}`);
console.log(` - Stars: ${repoInfo.stargazers_count}`);
console.log(` - Forks: ${repoInfo.forks_count}`);
console.log(` - Default Branch: ${repoInfo.default_branch}`);
console.log(` - Size: ${repoInfo.size} KB`);
console.log(` - Updated: ${repoInfo.updated_at}`);
if (isPublic) {
console.log(`✅ Repository is accessible and public`);
// Try to analyze the codebase
console.log(`🔍 Analyzing codebase...`);
const analysis = await githubService.analyzeCodebase(owner, repo, repoInfo.default_branch, true);
console.log(`📈 Codebase analysis completed:`);
console.log(` - Total Files: ${analysis.total_files}`);
console.log(` - Languages: ${Object.keys(analysis.languages).join(', ')}`);
console.log(` - Main Language: ${analysis.main_language}`);
console.log(` - Structure: ${analysis.structure ? 'Available' : 'Not available'}`);
// Try to download the repository
console.log(`📥 Attempting to download repository...`);
const downloadResult = await githubService.downloadRepository(owner, repo, repoInfo.default_branch);
if (downloadResult.success) {
console.log(`✅ Repository downloaded successfully!`);
console.log(`📁 Local path: ${downloadResult.local_path}`);
console.log(`📊 Download stats:`);
console.log(` - Files: ${downloadResult.files_count}`);
console.log(` - Directories: ${downloadResult.directories_count}`);
console.log(` - Size: ${downloadResult.total_size_bytes} bytes`);
// List some files
if (fs.existsSync(downloadResult.local_path)) {
const files = fs.readdirSync(downloadResult.local_path);
console.log(`📄 Sample files in root:`);
files.slice(0, 10).forEach(file => {
const filePath = path.join(downloadResult.local_path, file);
const stat = fs.statSync(filePath);
console.log(` - ${file} (${stat.isDirectory() ? 'directory' : 'file'})`);
});
}
} else {
console.log(`❌ Repository download failed: ${downloadResult.error}`);
}
} else {
console.log(`🔒 Repository is private - authentication required`);
}
} catch (error) {
console.log(`❌ Error accessing repository: ${error.message}`);
if (error.status === 404) {
console.log(`🔍 Repository might be private or not found`);
}
}
} catch (error) {
console.error(`💥 Test failed: ${error.message}`);
console.error(error.stack);
}
}
// Run the test
testRepository();

70
services/git-integration/test-webhook.js
View File

@ -1,70 +0,0 @@
// test-webhook.js - Simple test script for webhook endpoint
const fetch = require('node-fetch');
const WEBHOOK_URL = 'http://localhost:8012/api/github/webhook';
// Test webhook with a sample GitHub push event
const testPayload = {
ref: 'refs/heads/main',
before: 'abc123',
after: 'def456',
repository: {
id: 123456,
name: 'test-repo',
full_name: 'testuser/test-repo',
owner: {
login: 'testuser',
id: 789
}
},
pusher: {
name: 'testuser',
email: 'test@example.com'
},
commits: [
{
id: 'def456',
message: 'Test commit',
author: {
name: 'Test User',
email: 'test@example.com'
}
}
]
};
async function testWebhook() {
try {
console.log('🧪 Testing webhook endpoint...');
console.log(`📡 Sending POST request to: ${WEBHOOK_URL}`);
const response = await fetch(WEBHOOK_URL, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-GitHub-Event': 'push',
'X-GitHub-Delivery': 'test-delivery-123'
},
body: JSON.stringify(testPayload)
});
const result = await response.json();
console.log('📊 Response Status:', response.status);
console.log('📋 Response Body:', JSON.stringify(result, null, 2));
if (response.ok) {
console.log('✅ Webhook test successful!');
} else {
console.log('❌ Webhook test failed!');
}
} catch (error) {
console.error('❌ Error testing webhook:', error.message);
}
}
// Run the test
testWebhook();