diff --git a/MICROSERVICES_DEPLOYMENT_GUIDE.md b/MICROSERVICES_DEPLOYMENT_GUIDE.md new file mode 100644 index 0000000..5319f2a --- /dev/null +++ b/MICROSERVICES_DEPLOYMENT_GUIDE.md @@ -0,0 +1,230 @@ +# Microservices Architecture Deployment Guide + +## **Architecture Overview** + +``` +Frontend (Next.js) → API Gateway (Express) → Git Integration Service (Node.js) + Port 3001 Port 8000 Port 8012 +``` + +## **✅ Changes Applied** + +### **1. Frontend Fixes** +- ✅ Replaced `fetch()` calls with `authApiClient` in `DiffViewerContext.tsx` +- ✅ Replaced `fetch()` calls with `authApiClient` in `diff-viewer/page.tsx` +- ✅ All API calls now go through API Gateway (port 8000) instead of frontend (port 3001) + +### **2. Backend Enhancements** +- ✅ Added missing `/api/github/repository/:id/resolve-path` endpoint +- ✅ Enhanced path resolution with case-insensitive file matching +- ✅ All existing endpoints remain functional + +### **3. API Gateway Routing** +- ✅ Added `/api/diffs` route to API Gateway +- ✅ Configured proper proxy forwarding to Git Integration Service +- ✅ Added authentication middleware for diff operations +- ✅ Set appropriate timeouts for diff operations (120 seconds) + +## **🚀 Deployment Steps** + +### **1. Start All Services** + +```bash +# Navigate to backend directory +cd /home/tech4biz/Documents/merge/codenuk-backend-live + +# Start all services with Docker Compose +docker-compose up -d + +# Check service status +docker-compose ps +``` + +### **2. Start Frontend** + +```bash +# Navigate to frontend directory +cd /home/tech4biz/Documents/merge/codenuk-frontend-live + +# Install dependencies (if not already done) +npm install + +# Start frontend development server +npm run dev +``` + +### **3. Verify Architecture** + +```bash +# Run the architecture test +cd /home/tech4biz/Documents/merge/codenuk-backend-live +node test-microservices-architecture.js +``` + +## **🔍 Service Endpoints** + +### **Frontend (Port 3001)** +- **URL**: http://localhost:3001 +- **Purpose**: Next.js React application +- **API Calls**: All go through API Gateway (port 8000) + +### **API Gateway (Port 8000)** +- **URL**: http://localhost:8000 +- **Purpose**: Single entry point for all backend services +- **Routes**: + - `/api/github/*` → Git Integration Service + - `/api/diffs/*` → Git Integration Service + - `/api/vcs/*` → Git Integration Service + - `/api/ai/*` → Git Integration Service + +### **Git Integration Service (Port 8012)** +- **URL**: http://localhost:8012 +- **Purpose**: Handles all Git operations +- **Routes**: + - `/api/github/*` - GitHub integration + - `/api/diffs/*` - Diff viewer operations + - `/api/vcs/*` - Multi-provider VCS support + - `/api/ai/*` - AI streaming operations + +## **🧪 Testing the Architecture** + +### **1. Test Service Health** + +```bash +# Test API Gateway +curl http://localhost:8000/health + +# Test Git Integration Service +curl http://localhost:8012/health + +# Test Frontend +curl http://localhost:3001 +``` + +### **2. Test API Routing** + +```bash +# Test GitHub endpoints through gateway +curl http://localhost:8000/api/github/health + +# Test diff endpoints through gateway +curl http://localhost:8000/api/diffs/repositories + +# Test direct service call (should work) +curl http://localhost:8012/api/github/health +``` + +### **3. Test Frontend Integration** + +1. Open http://localhost:3001 +2. Navigate to GitHub repositories page +3. Navigate to diff viewer page +4. Check browser network tab - all API calls should go to port 8000 + +## **🔧 Configuration** + +### **Environment Variables** + +#### **Frontend (.env.local)** +```env +NEXT_PUBLIC_API_URL=http://localhost:8000 +``` + +#### **API Gateway (.env)** +```env +GIT_INTEGRATION_URL=http://git-integration:8012 +PORT=8000 +``` + +#### **Git Integration Service (.env)** +```env +PORT=8012 +DATABASE_URL=postgresql://user:password@postgres:5432/codenuk +``` + +## **🐛 Troubleshooting** + +### **Common Issues** + +#### **1. Frontend calls port 3001 instead of 8000** +- **Cause**: Using `fetch()` instead of `authApiClient` +- **Fix**: Replace all `fetch('/api/...')` with `authApiClient.get('/api/...')` + +#### **2. API Gateway returns 502 errors** +- **Cause**: Git Integration Service not running +- **Fix**: Check `docker-compose ps` and restart services + +#### **3. CORS errors in browser** +- **Cause**: Frontend trying to call different ports +- **Fix**: Ensure all API calls go through port 8000 + +#### **4. Authentication errors** +- **Cause**: Missing or invalid JWT tokens +- **Fix**: Check authentication flow and token refresh + +### **Debug Commands** + +```bash +# Check service logs +docker-compose logs git-integration +docker-compose logs api-gateway + +# Check service connectivity +docker-compose exec api-gateway curl http://git-integration:8012/health + +# Test specific endpoints +curl -H "Authorization: Bearer " http://localhost:8000/api/github/user/repositories +``` + +## **📊 Monitoring** + +### **Service Health Checks** + +```bash +# API Gateway health +curl http://localhost:8000/health + +# Git Integration health +curl http://localhost:8012/health + +# Frontend health +curl http://localhost:3001 +``` + +### **Log Monitoring** + +```bash +# Follow all logs +docker-compose logs -f + +# Follow specific service logs +docker-compose logs -f git-integration +docker-compose logs -f api-gateway +``` + +## **🎯 Success Criteria** + +✅ **Frontend (3001)** - All API calls go to port 8000 +✅ **API Gateway (8000)** - Routes requests to appropriate services +✅ **Git Integration (8012)** - Handles all Git operations +✅ **Authentication** - JWT tokens properly forwarded +✅ **Error Handling** - Proper error responses and timeouts +✅ **CORS** - No cross-origin issues + +## **📈 Performance Considerations** + +- **API Gateway**: 200 requests/minute limit for diff operations +- **Git Integration**: 120-second timeout for large operations +- **Frontend**: 10-second timeout for API calls +- **Database**: Connection pooling enabled + +## **🔒 Security** + +- **Authentication**: JWT tokens required for write operations +- **Authorization**: User context forwarded to services +- **Rate Limiting**: Applied at API Gateway level +- **CORS**: Configured for frontend domain only + +--- + +**✅ Architecture is now properly configured for microservices deployment!** diff --git a/package-lock.json b/package-lock.json index 693bd69..e64ec59 100644 --- a/package-lock.json +++ b/package-lock.json @@ -6,7 +6,289 @@ "packages": { "": { "name": "codenuk-backend-live", - "version": "1.0.0" + "version": "1.0.0", + "dependencies": { + "axios": "^1.12.2" + } + }, + "node_modules/asynckit": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz", + "integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==", + "license": "MIT" + }, + "node_modules/axios": { + "version": "1.12.2", + "resolved": "https://registry.npmjs.org/axios/-/axios-1.12.2.tgz", + "integrity": "sha512-vMJzPewAlRyOgxV2dU0Cuz2O8zzzx9VYtbJOaBgXFeLc4IV/Eg50n4LowmehOOR61S8ZMpc2K5Sa7g6A4jfkUw==", + "license": "MIT", + "dependencies": { + "follow-redirects": "^1.15.6", + "form-data": "^4.0.4", + "proxy-from-env": "^1.1.0" + } + }, + "node_modules/call-bind-apply-helpers": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", + "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/combined-stream": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz", + "integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==", + "license": "MIT", + "dependencies": { + "delayed-stream": "~1.0.0" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/delayed-stream": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz", + "integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==", + "license": "MIT", + "engines": { + "node": ">=0.4.0" + } + }, + "node_modules/dunder-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", + "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.1", + "es-errors": "^1.3.0", + "gopd": "^1.2.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-define-property": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", + "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-object-atoms": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz", + "integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-set-tostringtag": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz", + "integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.6", + "has-tostringtag": "^1.0.2", + "hasown": "^2.0.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/follow-redirects": { + "version": "1.15.11", + "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.15.11.tgz", + "integrity": "sha512-deG2P0JfjrTxl50XGCDyfI97ZGVCxIpfKYmfyrQ54n5FO/0gfIES8C/Psl6kWVDolizcaaxZJnTS0QSMxvnsBQ==", + "funding": [ + { + "type": "individual", + "url": "https://github.com/sponsors/RubenVerborgh" + } + ], + "license": "MIT", + "engines": { + "node": ">=4.0" + }, + "peerDependenciesMeta": { + "debug": { + "optional": true + } + } + }, + "node_modules/form-data": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.4.tgz", + "integrity": "sha512-KrGhL9Q4zjj0kiUt5OO4Mr/A/jlI2jDYs5eHBpYHPcBEVSiipAvn2Ko2HnPe20rmcuuvMHNdZFp+4IlGTMF0Ow==", + "license": "MIT", + "dependencies": { + "asynckit": "^0.4.0", + "combined-stream": "^1.0.8", + "es-set-tostringtag": "^2.1.0", + "hasown": "^2.0.2", + "mime-types": "^2.1.12" + }, + "engines": { + "node": ">= 6" + } + }, + "node_modules/function-bind": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", + "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-intrinsic": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", + "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "es-define-property": "^1.0.1", + "es-errors": "^1.3.0", + "es-object-atoms": "^1.1.1", + "function-bind": "^1.1.2", + "get-proto": "^1.0.1", + "gopd": "^1.2.0", + "has-symbols": "^1.1.0", + "hasown": "^2.0.2", + "math-intrinsics": "^1.1.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", + "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", + "license": "MIT", + "dependencies": { + "dunder-proto": "^1.0.1", + "es-object-atoms": "^1.0.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/gopd": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", + "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/has-symbols": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", + "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/has-tostringtag": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz", + "integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==", + "license": "MIT", + "dependencies": { + "has-symbols": "^1.0.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/hasown": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz", + "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==", + "license": "MIT", + "dependencies": { + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/math-intrinsics": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", + "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/mime-db": { + "version": "1.52.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz", + "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime-types": { + "version": "2.1.35", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz", + "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==", + "license": "MIT", + "dependencies": { + "mime-db": "1.52.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/proxy-from-env": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz", + "integrity": "sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg==", + "license": "MIT" } } } diff --git a/package.json b/package.json index 3d6f9ef..4305782 100644 --- a/package.json +++ b/package.json @@ -4,7 +4,8 @@ "version": "1.0.0", "scripts": { "migrate:all": "bash scripts/migrate-all.sh" + }, + "dependencies": { + "axios": "^1.12.2" } } - - diff --git a/services/api-gateway/src/server.js b/services/api-gateway/src/server.js index a2c79c4..860672e 100644 --- a/services/api-gateway/src/server.js +++ b/services/api-gateway/src/server.js @@ -111,6 +111,7 @@ app.use('/api/tech-stack', express.json({ limit: '10mb' })); app.use('/api/features', express.json({ limit: '10mb' })); app.use('/api/admin', express.json({ limit: '10mb' })); app.use('/api/github', express.json({ limit: '10mb' })); +app.use('/api/diffs', express.json({ limit: '10mb' })); app.use('/api/mockup', express.json({ limit: '10mb' })); app.use('/api/ai', express.json({ limit: '10mb' })); app.use('/health', express.json({ limit: '10mb' })); @@ -394,7 +395,7 @@ app.use('/api/templates', 'Connection': 'keep-alive', // Forward user context from auth middleware 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }), 'Authorization': req.headers.authorization }, timeout: 8000, @@ -594,7 +595,7 @@ app.use('/api/unified', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 30000, validateStatus: () => true, @@ -732,7 +733,7 @@ app.use('/api/ai/analyze-feature', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 25000, validateStatus: () => true, @@ -768,6 +769,179 @@ app.use('/api/ai/analyze-feature', } ); +// AI Repository Analysis - Route for repository AI analysis + +console.log('🔧 Registering /api/ai/repository proxy route...'); +app.use('/api/ai/repository', + createServiceLimiter(200), + // Allow unauthenticated access for AI analysis + (req, res, next) => { + console.log(`🤖 [AI REPO ANALYSIS PROXY] ${req.method} ${req.originalUrl}`); + next(); + }, + (req, res, next) => { + const gitIntegrationUrl = serviceTargets.GIT_INTEGRATION_URL; + const targetUrl = `${gitIntegrationUrl}${req.originalUrl}`; + console.log(`🤖 [AI REPO ANALYSIS PROXY] ${req.method} ${req.originalUrl} → ${targetUrl}`); + + // Check if this is a streaming endpoint + const isStreamingEndpoint = req.originalUrl.includes('/ai-stream') || + req.originalUrl.includes('/diff-stream') || + req.originalUrl.includes('/diff-analysis') && req.query.stream === 'true'; + + if (isStreamingEndpoint) { + console.log(`🌊 [AI REPO ANALYSIS PROXY] Streaming endpoint detected - using direct pipe`); + + // For streaming endpoints, use direct HTTP pipe without buffering + const http = require('http'); + const url = require('url'); + const targetUrlObj = url.parse(targetUrl); + + const proxyReq = http.request({ + hostname: targetUrlObj.hostname, + port: targetUrlObj.port, + path: targetUrlObj.path, + method: req.method, + headers: { + 'Content-Type': 'application/json', + 'User-Agent': 'API-Gateway/1.0', + 'Connection': 'keep-alive', + 'X-User-ID': req.headers['x-user-id'] || req.user?.id || req.user?.userId, + 'Cache-Control': 'no-cache', + 'Transfer-Encoding': 'chunked', + 'X-Accel-Buffering': 'no', + ...(req.user?.role && { 'X-User-Role': req.user.role }), + ...(req.headers.authorization && { 'Authorization': req.headers.authorization }) + }, + timeout: 120000 + }, (proxyRes) => { + console.log(`🌊 [AI REPO ANALYSIS PROXY] Streaming response: ${proxyRes.statusCode}`); + + // Set streaming headers + res.setHeader('Content-Type', 'application/json'); + res.setHeader('Cache-Control', 'no-cache'); + res.setHeader('Connection', 'keep-alive'); + res.setHeader('Transfer-Encoding', 'chunked'); + res.setHeader('X-Accel-Buffering', 'no'); + res.setHeader('Access-Control-Allow-Origin', req.headers.origin || '*'); + res.setHeader('Access-Control-Allow-Credentials', 'true'); + res.status(proxyRes.statusCode); + + // Pipe the response directly without buffering + proxyRes.on('data', (chunk) => { + res.write(chunk); + // Force flush to ensure immediate transmission + if (res.flush) res.flush(); + }); + + proxyRes.on('end', () => { + res.end(); + }); + + proxyRes.on('error', (error) => { + console.error(`❌ [AI REPO ANALYSIS PROXY STREAMING PIPE ERROR]:`, error.message); + if (!res.headersSent) { + res.status(502).json({ + error: 'Streaming pipe error', + message: error.message, + service: 'git-integration' + }); + } + }); + }); + + proxyReq.on('error', (error) => { + console.error(`❌ [AI REPO ANALYSIS PROXY STREAMING ERROR]:`, error.message); + if (!res.headersSent) { + res.status(502).json({ + error: 'AI Repository Analysis service unavailable', + message: error.message, + service: 'git-integration', + target_url: targetUrl + }); + } + }); + + // Handle request body for POST/PUT/PATCH + if (req.method === 'POST' || req.method === 'PUT' || req.method === 'PATCH') { + const body = JSON.stringify(req.body || {}); + proxyReq.setHeader('Content-Length', Buffer.byteLength(body)); + proxyReq.write(body); + console.log(`📦 [AI REPO ANALYSIS PROXY] Request body:`, JSON.stringify(req.body)); + } + + proxyReq.end(); + + } else { + // For non-streaming endpoints, use the original axios approach + console.log(`📦 [AI REPO ANALYSIS PROXY] Non-streaming endpoint - using axios`); + + // Set response timeout to prevent hanging (increased for large responses) + res.setTimeout(120000, () => { + console.error('❌ [AI REPO ANALYSIS PROXY] Response timeout'); + if (!res.headersSent) { + res.status(504).json({ error: 'Gateway timeout', service: 'git-integration' }); + } + }); + + const options = { + method: req.method, + url: targetUrl, + headers: { + 'Content-Type': 'application/json', + 'User-Agent': 'API-Gateway/1.0', + 'Connection': 'keep-alive', + 'X-User-ID': req.headers['x-user-id'] || req.user?.id || req.user?.userId, + ...(req.user?.role && { 'X-User-Role': req.user.role }), + ...(req.headers.authorization && { 'Authorization': req.headers.authorization }) + }, + timeout: 110000, // Increased timeout for large responses + validateStatus: () => true, + maxRedirects: 0, + maxContentLength: 50 * 1024 * 1024, // 50MB max content length + maxBodyLength: 50 * 1024 * 1024 // 50MB max body length + }; + + // Always include request body for POST/PUT/PATCH requests + if (req.method === 'POST' || req.method === 'PUT' || req.method === 'PATCH') { + options.data = req.body || {}; + console.log(`📦 [AI REPO ANALYSIS PROXY] Request body:`, JSON.stringify(req.body)); + } + + axios(options) + .then(response => { + console.log(`✅ [AI REPO ANALYSIS PROXY] Response: ${response.status} for ${req.method} ${req.originalUrl}`); + console.log(`📊 [AI REPO ANALYSIS PROXY] Response size: ${JSON.stringify(response.data).length} bytes`); + if (!res.headersSent) { + // Set proper headers for large responses + res.setHeader('Content-Type', 'application/json'); + res.setHeader('Access-Control-Allow-Origin', req.headers.origin || '*'); + res.setHeader('Access-Control-Allow-Credentials', 'true'); + res.status(response.status).json(response.data); + } + }) + .catch(error => { + console.error(`❌ [AI REPO ANALYSIS PROXY ERROR]:`, error.message); + console.error(`❌ [AI REPO ANALYSIS PROXY ERROR CODE]:`, error.code); + console.error(`❌ [AI REPO ANALYSIS PROXY ERROR STACK]:`, error.stack); + if (!res.headersSent) { + if (error.response) { + console.log(`📊 [AI REPO ANALYSIS PROXY] Error response status: ${error.response.status}`); + res.status(error.response.status).json(error.response.data); + } else { + res.status(502).json({ + error: 'AI Repository Analysis service unavailable', + message: error.code || error.message, + service: 'git-integration', + target_url: targetUrl + }); + } + } + }); + } + } +); + // Template Manager AI - expose AI recommendations through the gateway console.log('🔧 Registering /api/ai/tech-stack proxy route...'); app.use('/api/ai/tech-stack', @@ -808,7 +982,7 @@ app.use('/api/ai/tech-stack', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 25000, validateStatus: () => true, @@ -883,7 +1057,7 @@ app.use('/api/questions', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 25000, validateStatus: () => true, @@ -1005,7 +1179,7 @@ app.use('/api/unison', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 25000, validateStatus: () => true, @@ -1068,7 +1242,7 @@ app.use('/api/recommendations', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 25000, validateStatus: () => true, @@ -1122,7 +1296,7 @@ app.post('/api/recommendations', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 25000, validateStatus: () => true, @@ -1178,7 +1352,7 @@ app.use('/ai/recommendations', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 25000, validateStatus: () => true, @@ -1220,6 +1394,14 @@ app.get('/api/test', (req, res) => { res.json({ success: true, message: 'Test route working' }); }); +// Debug route for bulk-analysis +console.log('🔧 Registering debug route for bulk-analysis...'); +app.post('/api/ai/repository/:id/bulk-analysis', (req, res) => { + console.log(`🔍 [DEBUG] Received bulk-analysis request for ${req.params.id}`); + console.log(`🔍 [DEBUG] Request body:`, req.body); + res.json({ success: true, message: 'Debug route working', repository_id: req.params.id, body: req.body }); +}); + // Features (Template Manager) - expose /api/features via gateway console.log('🔧 Registering /api/features proxy route...'); app.use('/api/features', @@ -1249,7 +1431,7 @@ app.use('/api/features', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 10000, validateStatus: () => true, @@ -1363,7 +1545,7 @@ app.use('/api/github', 'Connection': 'keep-alive', // Forward user context from auth middleware 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }), 'Authorization': req.headers.authorization, // Forward session and cookie data for OAuth flows 'Cookie': req.headers.cookie, @@ -1472,6 +1654,101 @@ app.use('/api/github', } ); +// Diff Viewer Service - Direct HTTP forwarding for diff operations +console.log('🔧 Registering /api/diffs proxy route...'); +app.use('/api/diffs', + createServiceLimiter(200), + // Debug: Log all requests to /api/diffs + (req, res, next) => { + console.log(`🚀 [DIFFS PROXY ENTRY] ${req.method} ${req.originalUrl}`); + next(); + }, + // Allow unauthenticated access for read-only requests + (req, res, next) => { + const url = req.originalUrl || ''; + console.log(`🔍 [DIFFS PROXY AUTH] ${req.method} ${url}`); + + // Allow unauthenticated access for read-only requests + if (req.method === 'GET') { + console.log(`✅ [DIFFS PROXY AUTH] GET request - using optional auth`); + return authMiddleware.verifyTokenOptional(req, res, () => authMiddleware.forwardUserContext(req, res, next)); + } + + // Require authentication for write operations + return authMiddleware.verifyToken(req, res, () => authMiddleware.forwardUserContext(req, res, next)); + }, + (req, res, next) => { + const gitServiceUrl = serviceTargets.GIT_INTEGRATION_URL; + console.log(`🔥 [DIFFS PROXY] ${req.method} ${req.originalUrl} → ${gitServiceUrl}${req.originalUrl}`); + + // Set timeout for diff operations + res.setTimeout(120000, () => { + console.error('❌ [DIFFS PROXY] Response timeout'); + if (!res.headersSent) { + res.status(504).json({ error: 'Gateway timeout', service: 'git-integration' }); + } + }); + + const options = { + method: req.method, + url: `${gitServiceUrl}${req.originalUrl}`, + headers: { + 'Content-Type': 'application/json', + 'User-Agent': 'API-Gateway/1.0', + 'X-Forwarded-For': req.ip, + 'X-Forwarded-Proto': req.protocol, + 'X-Forwarded-Host': req.get('host'), + 'Authorization': req.headers.authorization, + 'x-user-id': req.headers['x-user-id'] + }, + data: req.body, + timeout: 120000, + validateStatus: () => true, + maxRedirects: 0 + }; + + axios(options) + .then(response => { + console.log(`✅ [DIFFS PROXY] ${response.status} ${req.method} ${req.originalUrl}`); + + // Handle redirects + if (response.status >= 300 && response.status < 400 && response.headers.location) { + const location = response.headers.location; + console.log(`↪️ [DIFFS PROXY] Forwarding redirect to ${location}`); + + // Update redirect URL to use gateway port if it points to git-integration service + let updatedLocation = location; + if (location.includes('localhost:8012')) { + updatedLocation = location.replace('localhost:8012', 'localhost:8000'); + } + + res.redirect(response.status, updatedLocation); + return; + } + + // Forward response + res.status(response.status).json(response.data); + }) + .catch(error => { + console.error(`❌ [DIFFS PROXY] Error:`, error.message); + + if (error.code === 'ECONNREFUSED' || error.code === 'ENOTFOUND') { + res.status(502).json({ + error: 'Git integration service unavailable', + message: error.code || error.message, + service: 'git-integration' + }); + } else { + res.status(500).json({ + error: 'Internal server error', + message: error.message, + service: 'git-integration' + }); + } + }); + } +); + // VCS Integration Service - Direct HTTP forwarding for Bitbucket, GitLab, Gitea console.log('🔧 Registering /api/vcs proxy route...'); app.use('/api/vcs', @@ -1522,7 +1799,7 @@ app.use('/api/vcs', 'Connection': 'keep-alive', // Forward user context from auth middleware 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }), 'Authorization': req.headers.authorization, // Forward session and cookie data for OAuth flows 'Cookie': req.headers.cookie, @@ -1651,7 +1928,7 @@ app.use('/api/mockup', 'Connection': 'keep-alive', 'Authorization': req.headers.authorization, 'X-User-ID': req.user?.id || req.user?.userId, - 'X-User-Role': req.user?.role, + ...(req.user?.role && { 'X-User-Role': req.user.role }) }, timeout: 25000, validateStatus: () => true, diff --git a/services/code-generator/Dockerfile b/services/code-generator/Dockerfile index 24a4898..abb05a6 100644 --- a/services/code-generator/Dockerfile +++ b/services/code-generator/Dockerfile @@ -9,7 +9,7 @@ RUN apt-get update && apt-get install -y \ # Copy requirements and install Python dependencies COPY requirements.txt . -RUN pip install --no-cache-dir -r requirements.txt +RUN pip install -r requirements.txt # Copy application code COPY src/ ./src/ diff --git a/services/git-integration.zip b/services/git-integration.zip index 563c784..d619315 100644 Binary files a/services/git-integration.zip and b/services/git-integration.zip differ diff --git a/services/git-integration/src/routes/ai-streaming.routes.js b/services/git-integration/src/routes/ai-streaming.routes.js index fd05efb..ee8073e 100644 --- a/services/git-integration/src/routes/ai-streaming.routes.js +++ b/services/git-integration/src/routes/ai-streaming.routes.js @@ -10,6 +10,9 @@ const aiStreamingService = new AIStreamingService(); router.get('/repository/:id/ai-stream', async (req, res) => { try { const { id: repositoryId } = req.params; + const userId = req.headers['x-user-id'] || req.headers['X-User-ID'] || req.headers['X-USER-ID']; + + const { file_types = 'auto', // Auto-detect all file types max_size = 3000000, // Increased to 3MB to include larger files @@ -19,6 +22,14 @@ router.get('/repository/:id/ai-stream', async (req, res) => { chunk_size = 'auto' // Auto-calculate optimal chunk size } = req.query; + // Validate user ID + if (!userId) { + return res.status(400).json({ + success: false, + message: 'User ID is required for AI analysis' + }); + } + // Validate repository exists const repoQuery = 'SELECT id, repository_name FROM all_repositories WHERE id = $1'; const repoResult = await database.query(repoQuery, [repositoryId]); @@ -78,16 +89,19 @@ router.get('/repository/:id/ai-stream', async (req, res) => { // Calculate total chunks const totalChunks = Math.ceil(totalFiles / chunkSize); - // Create streaming session + // Create streaming session with user ID const sessionId = aiStreamingService.createStreamingSession(repositoryId, { fileTypes: fileTypesArray, maxSize: maxSizeBytes, includeBinary: includeBinaryFiles, directoryFilter: directory_filter, excludePatterns: excludePatternsArray, - chunkSize: chunkSize + chunkSize: chunkSize, + userId: userId }); + console.log(`🔍 [AI-Stream] Starting analysis for user ${userId}, repository ${repositoryId}, session ${sessionId}`); + // Update session with total info aiStreamingService.updateStreamingSession(sessionId, { totalFiles, @@ -98,16 +112,19 @@ router.get('/repository/:id/ai-stream', async (req, res) => { // Get repository info const repositoryInfo = await aiStreamingService.getRepositoryInfo(repositoryId); - // Set headers for streaming + // Set headers for streaming with proper format for line-by-line display res.setHeader('Content-Type', 'application/json'); res.setHeader('Cache-Control', 'no-cache'); res.setHeader('Connection', 'keep-alive'); + res.setHeader('Transfer-Encoding', 'chunked'); + res.setHeader('X-Accel-Buffering', 'no'); // Disable nginx buffering res.setHeader('X-Streaming-Session-ID', sessionId); // Send initial response res.write(JSON.stringify({ success: true, session_id: sessionId, + user_id: userId, repository_info: { id: repositoryInfo.id, name: repositoryInfo.name, @@ -165,19 +182,20 @@ router.get('/repository/:id/ai-stream', async (req, res) => { status: 'streaming' }); - // Send chunk data - res.write(JSON.stringify({ - type: 'chunk', - chunk_data: chunkResult, - progress: { - current_chunk: currentChunk + 1, - total_chunks: totalChunks, - processed_files: processedFiles, - total_files: totalFiles, - percentage: Math.round((processedFiles / totalFiles) * 100) - }, - timestamp: new Date().toISOString() - }) + '\n'); + // Send chunk data + res.write(JSON.stringify({ + type: 'chunk', + user_id: userId, + chunk_data: chunkResult, + progress: { + current_chunk: currentChunk + 1, + total_chunks: totalChunks, + processed_files: processedFiles, + total_files: totalFiles, + percentage: Math.round((processedFiles / totalFiles) * 100) + }, + timestamp: new Date().toISOString() + }) + '\n'); currentChunk++; @@ -190,6 +208,7 @@ router.get('/repository/:id/ai-stream', async (req, res) => { // Send error for this chunk res.write(JSON.stringify({ type: 'error', + user_id: userId, chunk_number: currentChunk + 1, error: error.message, timestamp: new Date().toISOString() @@ -202,6 +221,7 @@ router.get('/repository/:id/ai-stream', async (req, res) => { // Send completion message res.write(JSON.stringify({ type: 'complete', + user_id: userId, session_id: sessionId, total_files_processed: processedFiles, total_chunks_processed: currentChunk, @@ -541,6 +561,7 @@ router.get('/repository/:id/commit/:commitId/diff-analysis', async (req, res) => // Send initial response res.write(JSON.stringify({ ...response, + user_id: req.headers['x-user-id'], stream_status: 'started' }) + '\n'); @@ -549,6 +570,7 @@ router.get('/repository/:id/commit/:commitId/diff-analysis', async (req, res) => for (const chunk of chunks) { res.write(JSON.stringify({ type: 'analysis_chunk', + user_id: req.headers['x-user-id'], chunk_data: chunk, timestamp: new Date().toISOString() }) + '\n'); @@ -557,6 +579,7 @@ router.get('/repository/:id/commit/:commitId/diff-analysis', async (req, res) => // Send completion res.write(JSON.stringify({ type: 'complete', + user_id: req.headers['x-user-id'], stream_status: 'finished', total_chunks: chunks.length, timestamp: new Date().toISOString() @@ -676,16 +699,19 @@ router.get('/repository/:id/diff-stream', async (req, res) => { includeContext: include_context === 'true' }); - // Set headers for streaming + // Set headers for streaming with proper format for line-by-line display res.setHeader('Content-Type', 'application/json'); res.setHeader('Cache-Control', 'no-cache'); res.setHeader('Connection', 'keep-alive'); + res.setHeader('Transfer-Encoding', 'chunked'); + res.setHeader('X-Accel-Buffering', 'no'); // Disable nginx buffering res.setHeader('X-Streaming-Session-ID', sessionId); // Send initial response res.write(JSON.stringify({ success: true, session_id: sessionId, + user_id: req.headers['x-user-id'], repository_id: repositoryId, diff_id: diff_id, status: 'ready' @@ -700,6 +726,7 @@ router.get('/repository/:id/diff-stream', async (req, res) => { for (const chunk of analysisResult.chunks) { res.write(JSON.stringify({ type: 'analysis_chunk', + user_id: req.headers['x-user-id'], chunk_data: chunk, timestamp: new Date().toISOString() }) + '\n'); @@ -708,6 +735,7 @@ router.get('/repository/:id/diff-stream', async (req, res) => { // Send completion message res.write(JSON.stringify({ type: 'complete', + user_id: req.headers['x-user-id'], session_id: sessionId, total_chunks: analysisResult.chunks.length, processing_time_ms: analysisResult.processingTime, @@ -734,6 +762,9 @@ router.get('/repository/:id/diff-stream', async (req, res) => { router.post('/repository/:id/bulk-analysis', async (req, res) => { try { const { id: repositoryId } = req.params; + const userId = req.headers['x-user-id'] || req.headers['X-User-ID'] || req.headers['X-USER-ID']; + + const { commit_ids = [], analysis_type = 'bulk', @@ -741,7 +772,18 @@ router.post('/repository/:id/bulk-analysis', async (req, res) => { stream = 'false' } = req.body; - console.log(`🔍 Bulk analysis for repository ${repositoryId} with ${commit_ids.length} commits`); + // Validate user ID + if (!userId) { + return res.status(400).json({ + success: false, + message: 'User ID is required for AI analysis' + }); + } + + console.log(`🔍 [Bulk-Analysis] Starting analysis for user ${userId}, repository ${repositoryId} with ${commit_ids.length} commits`); + console.log(`🔍 [Bulk-Analysis] Commit IDs:`, commit_ids); + console.log(`🔍 [Bulk-Analysis] Analysis type:`, analysis_type); + console.log(`🔍 [Bulk-Analysis] Include content:`, include_content); // Validate repository exists const repoQuery = 'SELECT id, repository_name, owner_name FROM all_repositories WHERE id = $1'; @@ -776,44 +818,66 @@ router.post('/repository/:id/bulk-analysis', async (req, res) => { } // Get bulk commit details + console.log(`🔍 [Bulk-Analysis] Getting bulk commit details for ${commit_ids.length} commits...`); const commitResults = await aiStreamingService.getBulkCommitDetails(commit_ids); + console.log(`🔍 [Bulk-Analysis] Found ${commitResults.length} commit results`); // Read diff files in batch + console.log(`🔍 [Bulk-Analysis] Reading diff files in batch...`); const enrichedResults = await aiStreamingService.batchReadDiffFiles(commitResults); + console.log(`🔍 [Bulk-Analysis] Enriched ${enrichedResults.length} results`); // Get analysis summary + console.log(`🔍 [Bulk-Analysis] Getting analysis summary...`); const summary = await aiStreamingService.getBulkAnalysisSummary(enrichedResults); + console.log(`🔍 [Bulk-Analysis] Summary:`, summary); // Process for AI analysis + console.log(`🔍 [Bulk-Analysis] Processing for AI analysis...`); const aiInputs = await aiStreamingService.processBulkCommitsForAI(enrichedResults); + console.log(`🔍 [Bulk-Analysis] Generated ${aiInputs.length} AI inputs`); // Prepare response - only include ai_inputs to avoid duplication const response = { success: true, repository_id: repositoryId, + user_id: userId, analysis_type: analysis_type, summary: summary, ai_ready_commits: aiInputs.length, ai_inputs: include_content === 'true' ? aiInputs : [] }; + console.log(`🔍 [Bulk-Analysis] Preparing response with user_id: ${userId}`); + console.log(`🔍 [Bulk-Analysis] Response size: ${JSON.stringify(response).length} characters`); + // Handle streaming if requested if (stream === 'true') { + console.log(`🔍 [Bulk-Analysis] Sending streaming response...`); res.setHeader('Content-Type', 'application/json'); res.setHeader('Cache-Control', 'no-cache'); res.setHeader('Connection', 'keep-alive'); res.write(JSON.stringify(response) + '\n'); res.end(); } else { + console.log(`🔍 [Bulk-Analysis] Sending JSON response...`); res.json(response); } } catch (error) { - console.error('Error in bulk analysis endpoint:', error); + console.error('❌ [Bulk-Analysis] Error in bulk analysis endpoint:', error); + console.error('❌ [Bulk-Analysis] Error stack:', error.stack); + console.error('❌ [Bulk-Analysis] Request details:', { + repositoryId: req.params.id, + userId: req.headers['x-user-id'], + body: req.body + }); + res.status(500).json({ success: false, message: error.message || 'Failed to perform bulk analysis', - repository_id: req.params.id + repository_id: req.params.id, + user_id: req.headers['x-user-id'] || 'unknown' }); } }); diff --git a/services/git-integration/src/routes/ai-streaming.routes.js.backup2 b/services/git-integration/src/routes/ai-streaming.routes.js.backup2 new file mode 100644 index 0000000..fd05efb --- /dev/null +++ b/services/git-integration/src/routes/ai-streaming.routes.js.backup2 @@ -0,0 +1,1027 @@ +// routes/ai-streaming.routes.js +const express = require('express'); +const router = express.Router(); +const AIStreamingService = require('../services/ai-streaming.service'); +const database = require('../config/database'); + +const aiStreamingService = new AIStreamingService(); + +// WebSocket streaming endpoint for AI analysis +router.get('/repository/:id/ai-stream', async (req, res) => { + try { + const { id: repositoryId } = req.params; + const { + file_types = 'auto', // Auto-detect all file types + max_size = 3000000, // Increased to 3MB to include larger files + include_binary = 'true', // Changed to true to include binary files + directory_filter = '', + exclude_patterns = 'node_modules,dist,build,.git,coverage', + chunk_size = 'auto' // Auto-calculate optimal chunk size + } = req.query; + + // Validate repository exists + const repoQuery = 'SELECT id, repository_name FROM all_repositories WHERE id = $1'; + const repoResult = await database.query(repoQuery, [repositoryId]); + + if (repoResult.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not found' + }); + } + + // Parse parameters with auto-detection + let fileTypesArray; + let chunkSize; + + if (file_types === 'auto') { + // Auto-detect file types from repository + fileTypesArray = await aiStreamingService.getAvailableFileTypes(repositoryId); + } else { + fileTypesArray = file_types.split(',').map(t => t.trim()); + } + + if (chunk_size === 'auto') { + // Auto-calculate optimal chunk size based on total files + const totalFiles = await aiStreamingService.getRepositoryFilesCount(repositoryId, { + fileTypes: fileTypesArray, + maxSize: parseInt(max_size), + includeBinary: include_binary === 'true', + directoryFilter: directory_filter, + excludePatterns: exclude_patterns.split(',').map(p => p.trim()) + }); + chunkSize = aiStreamingService.calculateOptimalChunkSize(totalFiles); + } else { + chunkSize = parseInt(chunk_size); + } + + const maxSizeBytes = parseInt(max_size); + const includeBinaryFiles = include_binary === 'true'; + const excludePatternsArray = exclude_patterns.split(',').map(p => p.trim()); + + // Get total files count + const totalFiles = await aiStreamingService.getRepositoryFilesCount(repositoryId, { + fileTypes: fileTypesArray, + maxSize: maxSizeBytes, + includeBinary: includeBinaryFiles, + directoryFilter: directory_filter, + excludePatterns: excludePatternsArray + }); + + if (totalFiles === 0) { + return res.status(404).json({ + success: false, + message: 'No files found matching the criteria' + }); + } + + // Calculate total chunks + const totalChunks = Math.ceil(totalFiles / chunkSize); + + // Create streaming session + const sessionId = aiStreamingService.createStreamingSession(repositoryId, { + fileTypes: fileTypesArray, + maxSize: maxSizeBytes, + includeBinary: includeBinaryFiles, + directoryFilter: directory_filter, + excludePatterns: excludePatternsArray, + chunkSize: chunkSize + }); + + // Update session with total info + aiStreamingService.updateStreamingSession(sessionId, { + totalFiles, + totalChunks, + status: 'ready' + }); + + // Get repository info + const repositoryInfo = await aiStreamingService.getRepositoryInfo(repositoryId); + + // Set headers for streaming + res.setHeader('Content-Type', 'application/json'); + res.setHeader('Cache-Control', 'no-cache'); + res.setHeader('Connection', 'keep-alive'); + res.setHeader('X-Streaming-Session-ID', sessionId); + + // Send initial response + res.write(JSON.stringify({ + success: true, + session_id: sessionId, + repository_info: { + id: repositoryInfo.id, + name: repositoryInfo.name, + full_name: repositoryInfo.full_name, + description: repositoryInfo.description, + language: repositoryInfo.language, + size: repositoryInfo.size, + local_path: repositoryInfo.local_path + }, + streaming_config: { + total_files: totalFiles, + total_chunks: totalChunks, + chunk_size: chunkSize, + file_types: fileTypesArray, + max_size_bytes: maxSizeBytes, + include_binary: includeBinaryFiles, + directory_filter: directory_filter, + exclude_patterns: excludePatternsArray + }, + status: 'ready' + }) + '\n'); + + // Process files in chunks + let currentChunk = 0; + let processedFiles = 0; + + while (currentChunk < totalChunks) { + try { + const offset = currentChunk * chunkSize; + const files = await aiStreamingService.getFilesChunk(repositoryId, offset, chunkSize, { + fileTypes: fileTypesArray, + maxSize: maxSizeBytes, + includeBinary: includeBinaryFiles, + directoryFilter: directory_filter, + excludePatterns: excludePatternsArray + }); + + if (files.length === 0) { + break; + } + + // Process chunk + const chunkResult = await aiStreamingService.processFilesChunk( + files, + currentChunk + 1, + totalChunks + ); + + processedFiles += chunkResult.files_processed; + + // Update session + aiStreamingService.updateStreamingSession(sessionId, { + currentChunk: currentChunk + 1, + processedFiles, + status: 'streaming' + }); + + // Send chunk data + res.write(JSON.stringify({ + type: 'chunk', + chunk_data: chunkResult, + progress: { + current_chunk: currentChunk + 1, + total_chunks: totalChunks, + processed_files: processedFiles, + total_files: totalFiles, + percentage: Math.round((processedFiles / totalFiles) * 100) + }, + timestamp: new Date().toISOString() + }) + '\n'); + + currentChunk++; + + // Small delay to prevent overwhelming the client + await new Promise(resolve => setTimeout(resolve, 50)); + + } catch (error) { + console.error(`Error processing chunk ${currentChunk + 1}:`, error); + + // Send error for this chunk + res.write(JSON.stringify({ + type: 'error', + chunk_number: currentChunk + 1, + error: error.message, + timestamp: new Date().toISOString() + }) + '\n'); + + currentChunk++; + } + } + + // Send completion message + res.write(JSON.stringify({ + type: 'complete', + session_id: sessionId, + total_files_processed: processedFiles, + total_chunks_processed: currentChunk, + processing_time_ms: Date.now() - aiStreamingService.getStreamingSession(sessionId).startTime, + timestamp: new Date().toISOString() + }) + '\n'); + + // Clean up session + aiStreamingService.removeStreamingSession(sessionId); + + res.end(); + + } catch (error) { + console.error('Error in AI streaming:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to stream files for AI analysis' + }); + } +}); + +// Get streaming session status +router.get('/streaming-session/:sessionId', (req, res) => { + try { + const { sessionId } = req.params; + const session = aiStreamingService.getStreamingSession(sessionId); + + if (!session) { + return res.status(404).json({ + success: false, + message: 'Streaming session not found' + }); + } + + res.json({ + success: true, + session: { + session_id: sessionId, + repository_id: session.repositoryId, + status: session.status, + current_chunk: session.currentChunk, + total_chunks: session.totalChunks, + total_files: session.totalFiles, + processed_files: session.processedFiles, + progress_percentage: session.totalFiles > 0 ? + Math.round((session.processedFiles / session.totalFiles) * 100) : 0, + start_time: new Date(session.startTime).toISOString(), + last_activity: new Date(session.lastActivity).toISOString() + } + }); + + } catch (error) { + console.error('Error getting streaming session:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to get streaming session status' + }); + } +}); + +// Cancel streaming session +router.delete('/streaming-session/:sessionId', (req, res) => { + try { + const { sessionId } = req.params; + const session = aiStreamingService.getStreamingSession(sessionId); + + if (!session) { + return res.status(404).json({ + success: false, + message: 'Streaming session not found' + }); + } + + aiStreamingService.removeStreamingSession(sessionId); + + res.json({ + success: true, + message: 'Streaming session cancelled successfully' + }); + + } catch (error) { + console.error('Error cancelling streaming session:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to cancel streaming session' + }); + } +}); + +// Get active streaming sessions +router.get('/streaming-sessions', (req, res) => { + try { + const sessions = Array.from(aiStreamingService.activeStreams.entries()).map(([sessionId, session]) => ({ + session_id: sessionId, + repository_id: session.repositoryId, + status: session.status, + current_chunk: session.currentChunk, + total_chunks: session.totalChunks, + total_files: session.totalFiles, + processed_files: session.processedFiles, + progress_percentage: session.totalFiles > 0 ? + Math.round((session.processedFiles / session.totalFiles) * 100) : 0, + start_time: new Date(session.startTime).toISOString(), + last_activity: new Date(session.lastActivity).toISOString() + })); + + res.json({ + success: true, + active_sessions: sessions.length, + sessions: sessions + }); + + } catch (error) { + console.error('Error getting streaming sessions:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to get streaming sessions' + }); + } +}); + +// Cleanup old sessions +router.post('/cleanup-sessions', (req, res) => { + try { + aiStreamingService.cleanupOldSessions(); + + res.json({ + success: true, + message: 'Old streaming sessions cleaned up successfully' + }); + + } catch (error) { + console.error('Error cleaning up sessions:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to cleanup sessions' + }); + } +}); + +// Debug endpoint to check diff content availability +router.get('/repository/:id/commit/:commitId/debug', async (req, res) => { + try { + const { id: repositoryId, commitId } = req.params; + + console.log(`🔍 Debugging diff content for repository ${repositoryId}, commit ${commitId}`); + + // Check if commit exists in database + const commitQuery = ` + SELECT + id, + repository_id, + commit_sha, + message, + author_name, + author_email, + committed_at, + created_at + FROM repository_commit_details + WHERE commit_sha = $1 + `; + + const commitResult = await database.query(commitQuery, [commitId]); + + if (commitResult.rows.length === 0) { + return res.json({ + success: false, + message: 'Commit not found in database', + repository_id: repositoryId, + commit_id: commitId, + debug_info: { + commit_exists: false, + diff_content_exists: false, + local_file_exists: false + } + }); + } + + const commit = commitResult.rows[0]; + + // Check if diff content exists in database + const diffQuery = ` + SELECT + id, + commit_id, + diff_header, + diff_size_bytes, + storage_type, + external_storage_path, + file_path, + change_type, + processing_status, + created_at + FROM diff_contents + WHERE commit_id = $1 + `; + + const diffResult = await database.query(diffQuery, [commit.id]); + + // Check if local file exists + const fs = require('fs'); + const path = require('path'); + const diffDir = process.env.DIFF_STORAGE_DIR || '/home/tech4biz/Desktop/today work/git-diff'; + const commitDir = path.join(diffDir, commitId); + const diffFilePath = path.join(diffDir, `${commitId}.diff`); + + // Check for commit directory structure (new format) + const commitDirExists = fs.existsSync(commitDir); + let localFileExists = false; + let localFiles = []; + + if (commitDirExists) { + const files = fs.readdirSync(commitDir); + localFiles = files.filter(file => file.endsWith('.diff')); + localFileExists = localFiles.length > 0; + } else { + // Check for single diff file (old format) + localFileExists = fs.existsSync(diffFilePath); + } + + // Check file changes + const fileQuery = ` + SELECT + id, + commit_id, + file_path, + change_type, + created_at + FROM repository_commit_files + WHERE commit_id = $1 + `; + + const fileResult = await database.query(fileQuery, [commit.id]); + + res.json({ + success: true, + repository_id: repositoryId, + commit_id: commitId, + debug_info: { + commit_exists: true, + commit_details: commit, + diff_content_exists: diffResult.rows.length > 0, + diff_contents: diffResult.rows, + local_file_exists: localFileExists, + commit_dir_exists: commitDirExists, + commit_dir_path: commitDir, + local_files: localFiles, + single_file_path: diffFilePath, + file_changes_count: fileResult.rows.length, + file_changes: fileResult.rows + } + }); + + } catch (error) { + console.error('Error in debug endpoint:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to debug diff content', + repository_id: req.params.id, + commit_id: req.params.commitId + }); + } +}); + +// Single endpoint for incremental diff analysis +router.get('/repository/:id/commit/:commitId/diff-analysis', async (req, res) => { + try { + const { id: repositoryId, commitId } = req.params; + const { + include_context = 'true', + analysis_type = 'auto', // 'auto', 'full', 'incremental' + stream = 'false' + } = req.query; + + console.log(`🔍 Analyzing diff for repository ${repositoryId}, commit ${commitId}`); + + // Validate repository exists + const repoQuery = 'SELECT id, repository_name, owner_name FROM all_repositories WHERE id = $1'; + const repoResult = await database.query(repoQuery, [repositoryId]); + + if (repoResult.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not found', + repository_id: repositoryId + }); + } + + const repository = repoResult.rows[0]; + const repositoryName = repository.repository_name; + const ownerName = repository.owner_name; + + // Get repository storage path + const storageQuery = 'SELECT local_path FROM repository_storage WHERE repository_id = $1'; + const storageResult = await database.query(storageQuery, [repositoryId]); + + if (storageResult.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not stored locally', + repository_id: repositoryId + }); + } + + const repoPath = storageResult.rows[0].local_path; + + // Check if diff content exists locally + const actualDiffContent = await aiStreamingService.getActualDiffContent(commitId); + + if (!actualDiffContent) { + return res.status(404).json({ + success: false, + message: 'No diff content found for this commit', + repository_id: repositoryId, + commit_id: commitId + }); + } + + // Get simple file changes (NO FILE CREATION) + const fileChanges = await aiStreamingService.getSimpleFileChanges(commitId); + + // Simple response + const response = { + success: true, + repository_id: repositoryId, + commit_id: commitId, + files: fileChanges, + diff_content: actualDiffContent + }; + + // Handle streaming if requested + if (stream === 'true') { + res.setHeader('Content-Type', 'application/json'); + res.setHeader('Cache-Control', 'no-cache'); + res.setHeader('Connection', 'keep-alive'); + + // Send initial response + res.write(JSON.stringify({ + ...response, + stream_status: 'started' + }) + '\n'); + + // Stream analysis chunks + const chunks = await aiStreamingService.streamAnalysisChunks(analysis, diffMetadata); + for (const chunk of chunks) { + res.write(JSON.stringify({ + type: 'analysis_chunk', + chunk_data: chunk, + timestamp: new Date().toISOString() + }) + '\n'); + } + + // Send completion + res.write(JSON.stringify({ + type: 'complete', + stream_status: 'finished', + total_chunks: chunks.length, + timestamp: new Date().toISOString() + }) + '\n'); + + res.end(); + } else { + res.json(response); + } + + } catch (error) { + console.error('Error in diff analysis:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to analyze diff', + repository_id: req.params.id, + commit_id: req.params.commitId + }); + } +}); + +// Get stored diffs for a repository +router.get('/repository/:id/diffs', async (req, res) => { + try { + const { id: repositoryId } = req.params; + const { limit = 10, offset = 0 } = req.query; + + // Validate repository exists + const repoQuery = 'SELECT id, repository_name FROM all_repositories WHERE id = $1'; + const repoResult = await database.query(repoQuery, [repositoryId]); + + if (repoResult.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not found' + }); + } + + const diffs = await aiStreamingService.getRepositoryDiffs(repositoryId, { + limit: parseInt(limit), + offset: parseInt(offset) + }); + + res.json({ + success: true, + repository_id: repositoryId, + diffs: diffs, + total: diffs.length + }); + + } catch (error) { + console.error('Error getting repository diffs:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to get repository diffs' + }); + } +}); + +// AI analysis for diff content +router.get('/repository/:id/diff-analysis', async (req, res) => { + try { + const { id: repositoryId } = req.params; + const { diff_id, include_context = 'true' } = req.query; + + // Validate repository exists + const repoQuery = 'SELECT id, repository_name FROM all_repositories WHERE id = $1'; + const repoResult = await database.query(repoQuery, [repositoryId]); + + if (repoResult.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not found' + }); + } + + // Get diff analysis + const analysis = await aiStreamingService.analyzeDiffContent(repositoryId, diff_id, { + includeContext: include_context === 'true' + }); + + res.json({ + success: true, + repository_id: repositoryId, + analysis: analysis + }); + + } catch (error) { + console.error('Error analyzing diff content:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to analyze diff content' + }); + } +}); + +// Stream diff analysis (similar to ai-stream but for diffs) +router.get('/repository/:id/diff-stream', async (req, res) => { + try { + const { id: repositoryId } = req.params; + const { diff_id, include_context = 'true' } = req.query; + + // Validate repository exists + const repoQuery = 'SELECT id, repository_name FROM all_repositories WHERE id = $1'; + const repoResult = await database.query(repoQuery, [repositoryId]); + + if (repoResult.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not found' + }); + } + + // Create streaming session for diff analysis + const sessionId = aiStreamingService.createDiffStreamingSession(repositoryId, { + diffId: diff_id, + includeContext: include_context === 'true' + }); + + // Set headers for streaming + res.setHeader('Content-Type', 'application/json'); + res.setHeader('Cache-Control', 'no-cache'); + res.setHeader('Connection', 'keep-alive'); + res.setHeader('X-Streaming-Session-ID', sessionId); + + // Send initial response + res.write(JSON.stringify({ + success: true, + session_id: sessionId, + repository_id: repositoryId, + diff_id: diff_id, + status: 'ready' + }) + '\n'); + + // Process diff analysis + const analysisResult = await aiStreamingService.streamDiffAnalysis(sessionId, repositoryId, diff_id, { + includeContext: include_context === 'true' + }); + + // Send analysis chunks + for (const chunk of analysisResult.chunks) { + res.write(JSON.stringify({ + type: 'analysis_chunk', + chunk_data: chunk, + timestamp: new Date().toISOString() + }) + '\n'); + } + + // Send completion message + res.write(JSON.stringify({ + type: 'complete', + session_id: sessionId, + total_chunks: analysisResult.chunks.length, + processing_time_ms: analysisResult.processingTime, + timestamp: new Date().toISOString() + }) + '\n'); + + // Clean up session + aiStreamingService.removeStreamingSession(sessionId); + + res.end(); + + } catch (error) { + console.error('Error in diff streaming:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to stream diff analysis' + }); + } +}); + +// ==================== BULK COMMIT ANALYSIS ENDPOINTS ==================== + +// Bulk commit analysis endpoint +router.post('/repository/:id/bulk-analysis', async (req, res) => { + try { + const { id: repositoryId } = req.params; + const { + commit_ids = [], + analysis_type = 'bulk', + include_content = 'true', + stream = 'false' + } = req.body; + + console.log(`🔍 Bulk analysis for repository ${repositoryId} with ${commit_ids.length} commits`); + + // Validate repository exists + const repoQuery = 'SELECT id, repository_name, owner_name FROM all_repositories WHERE id = $1'; + const repoResult = await database.query(repoQuery, [repositoryId]); + + if (repoResult.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not found', + repository_id: repositoryId + }); + } + + // Validate commit IDs + if (!Array.isArray(commit_ids) || commit_ids.length === 0) { + return res.status(400).json({ + success: false, + message: 'commit_ids must be a non-empty array', + repository_id: repositoryId + }); + } + + // Limit the number of commits to prevent overload + const maxCommits = 50; + if (commit_ids.length > maxCommits) { + return res.status(400).json({ + success: false, + message: `Maximum ${maxCommits} commits allowed per request`, + repository_id: repositoryId, + requested_commits: commit_ids.length + }); + } + + // Get bulk commit details + const commitResults = await aiStreamingService.getBulkCommitDetails(commit_ids); + + // Read diff files in batch + const enrichedResults = await aiStreamingService.batchReadDiffFiles(commitResults); + + // Get analysis summary + const summary = await aiStreamingService.getBulkAnalysisSummary(enrichedResults); + + // Process for AI analysis + const aiInputs = await aiStreamingService.processBulkCommitsForAI(enrichedResults); + + // Prepare response - only include ai_inputs to avoid duplication + const response = { + success: true, + repository_id: repositoryId, + analysis_type: analysis_type, + summary: summary, + ai_ready_commits: aiInputs.length, + ai_inputs: include_content === 'true' ? aiInputs : [] + }; + + // Handle streaming if requested + if (stream === 'true') { + res.setHeader('Content-Type', 'application/json'); + res.setHeader('Cache-Control', 'no-cache'); + res.setHeader('Connection', 'keep-alive'); + res.write(JSON.stringify(response) + '\n'); + res.end(); + } else { + res.json(response); + } + + } catch (error) { + console.error('Error in bulk analysis endpoint:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to perform bulk analysis', + repository_id: req.params.id + }); + } +}); + +// Get bulk analysis status +router.get('/repository/:id/bulk-analysis/status', async (req, res) => { + try { + const { id: repositoryId } = req.params; + const { commit_ids } = req.query; + + console.log(`📊 Getting bulk analysis status for repository ${repositoryId}`); + + // Validate repository exists + const repoQuery = 'SELECT id, repository_name FROM all_repositories WHERE id = $1'; + const repoResult = await database.query(repoQuery, [repositoryId]); + + if (repoResult.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not found', + repository_id: repositoryId + }); + } + + // Parse commit IDs from query string + const commitIds = commit_ids ? commit_ids.split(',') : []; + + if (commitIds.length === 0) { + return res.status(400).json({ + success: false, + message: 'commit_ids parameter is required', + repository_id: repositoryId + }); + } + + // Get status for each commit + const statusResults = []; + + for (const commitId of commitIds) { + try { + // Check if commit exists in database + const commitQuery = ` + SELECT id, commit_sha, message, committed_at + FROM repository_commit_details + WHERE id = $1 + `; + + const commitResult = await database.query(commitQuery, [commitId]); + + if (commitResult.rows.length === 0) { + statusResults.push({ + commitId: commitId, + status: 'not_found', + message: 'Commit not found in database' + }); + continue; + } + + // Check if diff content exists + const diffQuery = ` + SELECT COUNT(*) as count + FROM diff_contents + WHERE commit_id = $1 + `; + + const diffResult = await database.query(diffQuery, [commitId]); + const hasDiffContent = parseInt(diffResult.rows[0].count) > 0; + + statusResults.push({ + commitId: commitId, + status: 'found', + hasDiffContent: hasDiffContent, + commitDetails: commitResult.rows[0] + }); + + } catch (error) { + console.error(`Error checking status for commit ${commitId}:`, error); + statusResults.push({ + commitId: commitId, + status: 'error', + message: error.message + }); + } + } + + // Calculate summary + const summary = { + total_commits: statusResults.length, + found_commits: statusResults.filter(r => r.status === 'found').length, + not_found_commits: statusResults.filter(r => r.status === 'not_found').length, + error_commits: statusResults.filter(r => r.status === 'error').length, + commits_with_diff: statusResults.filter(r => r.hasDiffContent).length + }; + + res.json({ + success: true, + repository_id: repositoryId, + summary: summary, + commits: statusResults + }); + + } catch (error) { + console.error('Error in bulk analysis status endpoint:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to get bulk analysis status', + repository_id: req.params.id + }); + } +}); + +// Get available commits for bulk analysis +router.get('/repository/:id/commits', async (req, res) => { + try { + const { id: repositoryId } = req.params; + const { + limit = 100, + offset = 0, + has_diff_content = 'all' // 'all', 'true', 'false' + } = req.query; + + console.log(`📋 Getting available commits for repository ${repositoryId}`); + + // Validate repository exists + const repoQuery = 'SELECT id, repository_name FROM all_repositories WHERE id = $1'; + const repoResult = await database.query(repoQuery, [repositoryId]); + + if (repoResult.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not found', + repository_id: repositoryId + }); + } + + // Build query based on has_diff_content filter + let whereClause = 'WHERE rcd.repository_id = $1'; + let queryParams = [repositoryId]; + let paramIndex = 2; + + if (has_diff_content === 'true') { + whereClause += ` AND EXISTS ( + SELECT 1 FROM diff_contents dc + WHERE dc.commit_id = rcd.id + )`; + } else if (has_diff_content === 'false') { + whereClause += ` AND NOT EXISTS ( + SELECT 1 FROM diff_contents dc + WHERE dc.commit_id = rcd.id + )`; + } + + const query = ` + SELECT + rcd.id, + rcd.commit_sha, + rcd.message, + rcd.committed_at, + CASE + WHEN EXISTS (SELECT 1 FROM diff_contents dc WHERE dc.commit_id = rcd.id) + THEN true + ELSE false + END as has_diff_content, + ( + SELECT COUNT(*) + FROM repository_commit_files rcf + WHERE rcf.commit_id = rcd.id + ) as files_count + FROM repository_commit_details rcd + ${whereClause} + ORDER BY rcd.committed_at DESC + LIMIT $${paramIndex} OFFSET $${paramIndex + 1} + `; + + queryParams.push(parseInt(limit), parseInt(offset)); + + const result = await database.query(query, queryParams); + + // Get total count for pagination + const countQuery = ` + SELECT COUNT(*) as total + FROM repository_commit_details rcd + ${whereClause} + `; + + const countResult = await database.query(countQuery, [repositoryId]); + const totalCount = parseInt(countResult.rows[0].total); + + res.json({ + success: true, + repository_id: repositoryId, + commits: result.rows, + pagination: { + total: totalCount, + limit: parseInt(limit), + offset: parseInt(offset), + has_more: (parseInt(offset) + parseInt(limit)) < totalCount + } + }); + + } catch (error) { + console.error('Error in get commits endpoint:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to get commits', + repository_id: req.params.id + }); + } +}); + +module.exports = router; diff --git a/services/git-integration/src/routes/ai-streaming.service.js.backup b/services/git-integration/src/routes/ai-streaming.service.js.backup new file mode 100644 index 0000000..b682a61 --- /dev/null +++ b/services/git-integration/src/routes/ai-streaming.service.js.backup @@ -0,0 +1,2038 @@ +// services/ai-streaming.service.js +const fs = require('fs'); +const path = require('path'); +const database = require('../config/database'); +const FileStorageService = require('./file-storage.service'); + +class AIStreamingService { + constructor() { + this.fileStorageService = new FileStorageService(); + this.activeStreams = new Map(); // Track active streaming sessions + this.chunkSize = 200; // Files per chunk + this.maxFileSize = 1000000; // 1MB per file + this.maxContentLength = 500000; // 500KB content limit + } + + // Auto-detect available file types in repository + async getAvailableFileTypes(repositoryId) { + try { + const query = ` + SELECT DISTINCT file->>'file_extension' as file_extension + FROM repository_files rf, + jsonb_array_elements(rf.files) as file + WHERE rf.repository_id = $1 + AND file->>'file_extension' IS NOT NULL + ORDER BY file->>'file_extension' + `; + + const result = await database.query(query, [repositoryId]); + const fileTypes = result.rows.map(row => row.file_extension); + + // Include files without extensions (empty string or null) + const hasNoExtension = await database.query(` + SELECT COUNT(*) as count + FROM repository_files rf, + jsonb_array_elements(rf.files) as file + WHERE rf.repository_id = $1 + AND (file->>'file_extension' IS NULL OR file->>'file_extension' = '') + `, [repositoryId]); + + if (parseInt(hasNoExtension.rows[0].count) > 0) { + // Only add if not already present + if (!fileTypes.includes('')) { + fileTypes.push(''); // Add empty string for files without extensions + } + } + + console.log(`🔍 Auto-detected file types: ${fileTypes.join(', ')}`); + console.log(`📊 Total file types found: ${fileTypes.length}`); + return fileTypes; + + } catch (error) { + console.error('Error auto-detecting file types:', error); + // Fallback to common file types + return ['js', 'jsx', 'ts', 'tsx', 'py', 'java', 'json', 'md', 'txt', 'css', 'html', 'php', 'rb', 'go', 'rs', 'kt', 'swift', 'scala', 'clj', 'hs', 'elm', 'ml', 'fs', 'vb', 'pas', 'asm', 'sql', 'sh', 'bash', 'ps1', 'bat', 'cmd', 'xml', 'scss', 'sass', 'less', 'toml', 'ini', 'cfg', 'conf', 'env', 'rst', 'adoc', 'tex', 'r', 'm', 'pl', 'lua', 'dart', 'jl', 'nim', 'zig', 'v', 'd', 'cr', 'ex', 'exs']; + } + } + + // Calculate optimal chunk size based on total files + calculateOptimalChunkSize(totalFiles) { + if (totalFiles <= 50) { + return 10; // Small repos: 10 files per chunk + } else if (totalFiles <= 200) { + return 25; // Medium repos: 25 files per chunk + } else if (totalFiles <= 500) { + return 50; // Large repos: 50 files per chunk + } else if (totalFiles <= 1000) { + return 100; // Very large repos: 100 files per chunk + } else { + return 200; // Huge repos: 200 files per chunk + } + } + + // Get repository files count for streaming planning + async getRepositoryFilesCount(repositoryId, options = {}) { + const { + fileTypes = ['js', 'jsx', 'ts', 'tsx', 'py', 'java', 'json', 'md', 'txt', 'css', 'html', 'php', 'rb', 'go', 'rs', 'kt', 'swift', 'scala', 'clj', 'hs', 'elm', 'ml', 'fs', 'vb', 'pas', 'asm', 'sql', 'sh', 'bash', 'ps1', 'bat', 'cmd', 'xml', 'scss', 'sass', 'less', 'toml', 'ini', 'cfg', 'conf', 'env', 'rst', 'adoc', 'tex', 'r', 'm', 'pl', 'lua', 'dart', 'jl', 'nim', 'zig', 'v', 'd', 'cr', 'ex', 'exs'], + maxSize = 3000000, // Increased to 3MB + includeBinary = true, // Changed to true to include binary files + directoryFilter = '', + excludePatterns = ['node_modules', 'dist', 'build', '.git', 'coverage'] + } = options; + + try { + // Build conditions for JSONB query + let conditions = ['rf.repository_id = $1']; + const params = [repositoryId]; + let paramIndex = 2; + + // File type filter + if (fileTypes.length > 0) { + const typeConditions = []; + const normalizedFileTypes = []; + + for (const ext of fileTypes) { + if (ext === '') { + // Files without extensions + typeConditions.push(`(file->>'file_extension' IS NULL OR file->>'file_extension' = '')`); + } else { + // Files with extensions + const normalizedExt = ext.startsWith('.') ? ext : `.${ext}`; + typeConditions.push(`file->>'file_extension' = $${paramIndex + normalizedFileTypes.length}`); + normalizedFileTypes.push(normalizedExt); + } + } + + if (typeConditions.length > 0) { + conditions.push(`(${typeConditions.join(' OR ')})`); + params.push(...normalizedFileTypes); + paramIndex += normalizedFileTypes.length; + } + } + + // Size filter + if (maxSize > 0) { + conditions.push(`(file->>'file_size_bytes')::bigint <= $${paramIndex}`); + params.push(maxSize); + paramIndex++; + } + + // Binary filter + if (!includeBinary) { + conditions.push(`(file->>'is_binary')::boolean = false`); + } + + // Directory filter + if (directoryFilter) { + conditions.push(`rf.relative_path LIKE $${paramIndex}`); + params.push(`%${directoryFilter}%`); + paramIndex++; + } + + // Exclude patterns + if (excludePatterns.length > 0) { + const excludeConditions = excludePatterns.map((_, index) => + `file->>'relative_path' NOT LIKE $${paramIndex + index}` + ).join(' AND '); + conditions.push(`(${excludeConditions})`); + excludePatterns.forEach(pattern => { + params.push(`%${pattern}%`); + }); + paramIndex += excludePatterns.length; + } + + const query = ` + SELECT COUNT(*) as total_count + FROM repository_files rf, + jsonb_array_elements(rf.files) as file + WHERE ${conditions.join(' AND ')} + `; + + const result = await database.query(query, params); + return parseInt(result.rows[0].total_count); + + } catch (error) { + console.error('Error getting repository files count:', error); + return 0; + } + } + + // Get files for a specific chunk + async getFilesChunk(repositoryId, offset, limit, options = {}) { + const { + fileTypes = ['js', 'jsx', 'ts', 'tsx', 'py', 'java', 'json', 'md', 'txt', 'css', 'html', 'php', 'rb', 'go', 'rs', 'kt', 'swift', 'scala', 'clj', 'hs', 'elm', 'ml', 'fs', 'vb', 'pas', 'asm', 'sql', 'sh', 'bash', 'ps1', 'bat', 'cmd', 'xml', 'scss', 'sass', 'less', 'toml', 'ini', 'cfg', 'conf', 'env', 'rst', 'adoc', 'tex', 'r', 'm', 'pl', 'lua', 'dart', 'jl', 'nim', 'zig', 'v', 'd', 'cr', 'ex', 'exs'], + maxSize = 3000000, // Increased to 3MB + includeBinary = true, // Changed to true to include binary files + directoryFilter = '', + excludePatterns = ['node_modules', 'dist', 'build', '.git', 'coverage'] + } = options; + + try { + // Build conditions for JSONB query + let conditions = ['rf.repository_id = $1']; + const params = [repositoryId]; + let paramIndex = 2; + + // File type filter + if (fileTypes.length > 0) { + const typeConditions = []; + const normalizedFileTypes = []; + + for (const ext of fileTypes) { + if (ext === '') { + // Files without extensions + typeConditions.push(`(file->>'file_extension' IS NULL OR file->>'file_extension' = '')`); + } else { + // Files with extensions + const normalizedExt = ext.startsWith('.') ? ext : `.${ext}`; + typeConditions.push(`file->>'file_extension' = $${paramIndex + normalizedFileTypes.length}`); + normalizedFileTypes.push(normalizedExt); + } + } + + if (typeConditions.length > 0) { + conditions.push(`(${typeConditions.join(' OR ')})`); + params.push(...normalizedFileTypes); + paramIndex += normalizedFileTypes.length; + } + } + + // Size filter + if (maxSize > 0) { + conditions.push(`(file->>'file_size_bytes')::bigint <= $${paramIndex}`); + params.push(maxSize); + paramIndex++; + } + + // Binary filter + if (!includeBinary) { + conditions.push(`(file->>'is_binary')::boolean = false`); + } + + // Directory filter + if (directoryFilter) { + conditions.push(`rf.relative_path LIKE $${paramIndex}`); + params.push(`%${directoryFilter}%`); + paramIndex++; + } + + // Exclude patterns + if (excludePatterns.length > 0) { + const excludeConditions = excludePatterns.map((_, index) => + `file->>'relative_path' NOT LIKE $${paramIndex + index}` + ).join(' AND '); + conditions.push(`(${excludeConditions})`); + excludePatterns.forEach(pattern => { + params.push(`%${pattern}%`); + }); + paramIndex += excludePatterns.length; + } + + const query = ` + SELECT + COALESCE(rf.directory_id, rf.id) as id, + file->>'filename' as filename, + file->>'relative_path' as relative_path, + file->>'file_extension' as file_extension, + (file->>'file_size_bytes')::bigint as file_size_bytes, + file->>'mime_type' as mime_type, + (file->>'is_binary')::boolean as is_binary, + file->>'absolute_path' as absolute_path, + rf.relative_path as directory_path + FROM repository_files rf, + jsonb_array_elements(rf.files) as file + WHERE ${conditions.join(' AND ')} + ORDER BY file->>'relative_path' + LIMIT $${paramIndex} OFFSET $${paramIndex + 1} + `; + + params.push(limit, offset); + + const result = await database.query(query, params); + return result.rows; + + } catch (error) { + console.error('Error getting files chunk:', error); + throw error; + } + } + + // Read file content for AI analysis + async readFileContentForAI(filePath, isBinary) { + if (isBinary) { + return null; + } + + try { + // Check file size first + const stats = fs.statSync(filePath); + if (stats.size > this.maxFileSize) { + console.warn(`File ${filePath} is too large (${stats.size} bytes), skipping content`); + return null; + } + + // Read file content + const content = fs.readFileSync(filePath, 'utf8'); + + // Truncate if too long + if (content.length > this.maxContentLength) { + console.warn(`File ${filePath} content is too long (${content.length} chars), truncating`); + return content.substring(0, this.maxContentLength) + '\n\n... [Content truncated for performance]'; + } + + return content; + } catch (error) { + try { + // Fallback to Latin-1 + const content = fs.readFileSync(filePath, 'latin1'); + + if (content.length > this.maxContentLength) { + return content.substring(0, this.maxContentLength) + '\n\n... [Content truncated for performance]'; + } + + return content; + } catch (fallbackError) { + console.warn(`Could not read file ${filePath}:`, fallbackError.message); + return null; + } + } + } + + // Detect programming language + detectLanguage(fileExtension, content) { + const languageMap = { + '.js': 'javascript', '.ts': 'typescript', '.jsx': 'javascript', '.tsx': 'typescript', + '.vue': 'vue', '.py': 'python', '.java': 'java', '.cpp': 'cpp', '.c': 'c', + '.cs': 'csharp', '.php': 'php', '.rb': 'ruby', '.go': 'go', '.rs': 'rust', + '.kt': 'kotlin', '.swift': 'swift', '.scala': 'scala', '.clj': 'clojure', + '.hs': 'haskell', '.elm': 'elm', '.ml': 'ocaml', '.fs': 'fsharp', + '.vb': 'vbnet', '.html': 'html', '.css': 'css', '.scss': 'scss', + '.json': 'json', '.yaml': 'yaml', '.yml': 'yaml', '.xml': 'xml', + '.sql': 'sql', '.sh': 'bash', '.md': 'markdown' + }; + + return languageMap[fileExtension] || 'unknown'; + } + + // Detect framework + detectFramework(content, filename) { + if (!content) return null; + + const frameworkHints = { + 'react': /import.*from\s+['"]react['"]|React\.|useState|useEffect|jsx/i, + 'vue': /import.*from\s+['"]vue['"]|Vue\.|createApp|defineComponent/i, + 'angular': /@angular|@Component|@Injectable|@NgModule/i, + 'express': /require\s*\(\s*['"]express['"]|app\.get|app\.post|app\.use/i, + 'fastapi': /from\s+fastapi|@app\.|FastAPI\(/i, + 'django': /from\s+django|@csrf_exempt|HttpResponse/i, + 'spring': /@SpringBootApplication|@RestController|@Service|@Repository/i, + 'laravel': /use\s+Illuminate|Route::|@extends\s+['"]layouts/i + }; + + for (const [framework, pattern] of Object.entries(frameworkHints)) { + if (pattern.test(content)) { + return framework; + } + } + + return null; + } + + // Generate analysis hints + generateAnalysisHints(file, content, language) { + if (!content) return {}; + + const hints = { + is_component: false, + has_imports: false, + has_exports: false, + has_functions: false, + has_classes: false, + complexity_level: 'low', + framework_hints: [] + }; + + // Check for imports + hints.has_imports = /import\s+.*from|require\s*\(/.test(content); + + // Check for exports + hints.has_exports = /export\s+|module\.exports|exports\./.test(content); + + // Check for functions + hints.has_functions = /function\s+\w+|const\s+\w+\s*=\s*\(|def\s+\w+/.test(content); + + // Check for classes + hints.has_classes = /class\s+\w+|@Component|@Service/.test(content); + + // Check if it's a component (React, Vue, Angular) + if (language === 'javascript' || language === 'typescript') { + hints.is_component = /export\s+default|export\s+const.*=.*\(/i.test(content) && + (hints.has_imports || /jsx|tsx/.test(file.file_extension)); + } + + // Calculate complexity + const lines = content.split('\n').length; + const functions = (content.match(/function\s+\w+|const\s+\w+\s*=\s*\(|def\s+\w+/g) || []).length; + const classes = (content.match(/class\s+\w+/g) || []).length; + + if (lines > 500 || functions > 20 || classes > 10) { + hints.complexity_level = 'high'; + } else if (lines > 200 || functions > 10 || classes > 5) { + hints.complexity_level = 'medium'; + } + + return hints; + } + + // Process files chunk for AI analysis + async processFilesChunk(files, chunkNumber, totalChunks) { + const processedFiles = []; + const startTime = Date.now(); + + for (const file of files) { + try { + // Read file content + const content = await this.readFileContentForAI(file.absolute_path, file.is_binary); + + // Detect language and framework + const language = this.detectLanguage(file.file_extension, content); + const framework = this.detectFramework(content, file.filename); + + // Generate analysis hints + const analysisHints = this.generateAnalysisHints(file, content, language); + + const processedFile = { + id: file.id, + filename: file.filename, + relative_path: file.relative_path, + file_extension: file.file_extension, + file_size_bytes: file.file_size_bytes, + mime_type: file.mime_type, + language: language, + is_binary: file.is_binary, + line_count: content ? content.split('\n').length : 0, + char_count: content ? content.length : 0, + content: content, + analysis_hints: analysisHints, + framework: framework, + chunk_info: { + chunk_number: chunkNumber, + total_chunks: totalChunks, + file_index_in_chunk: processedFiles.length + 1, + total_files_in_chunk: files.length + } + }; + + processedFiles.push(processedFile); + + } catch (error) { + console.warn(`Error processing file ${file.relative_path}:`, error.message); + // Continue with other files + } + } + + const processingTime = Date.now() - startTime; + + return { + files: processedFiles, + chunk_number: chunkNumber, + total_chunks: totalChunks, + files_processed: processedFiles.length, + processing_time_ms: processingTime, + timestamp: new Date().toISOString() + }; + } + + // Create streaming session + createStreamingSession(repositoryId, options = {}) { + const sessionId = `stream_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`; + + this.activeStreams.set(sessionId, { + repositoryId, + options, + status: 'initializing', + currentChunk: 0, + totalChunks: 0, + totalFiles: 0, + processedFiles: 0, + startTime: Date.now(), + lastActivity: Date.now() + }); + + return sessionId; + } + + // Get streaming session + getStreamingSession(sessionId) { + return this.activeStreams.get(sessionId); + } + + // Update streaming session + updateStreamingSession(sessionId, updates) { + const session = this.activeStreams.get(sessionId); + if (session) { + Object.assign(session, updates); + session.lastActivity = Date.now(); + } + } + + // Remove streaming session + removeStreamingSession(sessionId) { + this.activeStreams.delete(sessionId); + } + + // Clean up old sessions (older than 1 hour) + cleanupOldSessions() { + const oneHourAgo = Date.now() - (60 * 60 * 1000); + + for (const [sessionId, session] of this.activeStreams.entries()) { + if (session.lastActivity < oneHourAgo) { + this.activeStreams.delete(sessionId); + console.log(`Cleaned up old streaming session: ${sessionId}`); + } + } + } + + // Get repository info for streaming + async getRepositoryInfo(repositoryId) { + try { + const query = ` + SELECT + gr.id, + gr.repository_name as name, + gr.repository_url, + gr.owner_name, + gr.branch_name, + gr.is_public, + gr.requires_auth, + gr.last_synced_at, + gr.sync_status, + gr.metadata, + gr.codebase_analysis, + gr.created_at, + gr.updated_at, + rs.local_path, + rs.storage_status + FROM all_repositories gr + LEFT JOIN repository_storage rs ON gr.id = rs.repository_id + WHERE gr.id = $1 + `; + + const result = await database.query(query, [repositoryId]); + + if (result.rows.length === 0) { + throw new Error('Repository not found'); + } + + const repo = result.rows[0]; + + // Create a full_name from owner_name and repository_name + const fullName = `${repo.owner_name}/${repo.repository_name}`; + + return { + ...repo, + full_name: fullName, + description: repo.metadata?.description || '', + language: repo.metadata?.language || 'Unknown', + size: repo.metadata?.size || 0, + stars_count: repo.metadata?.stargazers_count || 0, + forks_count: repo.metadata?.forks_count || 0 + }; + } catch (error) { + console.error('Error getting repository info:', error); + throw error; + } + } + + // Store diff content for AI analysis + async storeDiffContent(diffData) { + const { + repositoryId, + diffContent, + commitSha, + parentSha, + commitMessage, + authorName, + authorEmail, + commitDate, + fileChanges + } = diffData; + + try { + // First, create or get commit record + let commitId; + const commitQuery = ` + SELECT id FROM repository_commit_details + WHERE repository_id = $1 AND commit_sha = $2 + `; + const commitResult = await database.query(commitQuery, [repositoryId, commitSha]); + + if (commitResult.rows.length === 0) { + // Create new commit record + const insertCommitQuery = ` + INSERT INTO repository_commit_details ( + repository_id, commit_sha, message, + author_name, author_email, committed_at, created_at + ) VALUES ($1, $2, $3, $4, $5, $6, $7, NOW()) + RETURNING id + `; + const commitInsertResult = await database.query(insertCommitQuery, [ + repositoryId, commitSha, parentSha, commitMessage, + authorName, authorEmail, commitDate + ]); + commitId = commitInsertResult.rows[0].id; + } else { + commitId = commitResult.rows[0].id; + } + + // Store diff content + const diffId = require('crypto').randomUUID(); + const diffSize = Buffer.byteLength(diffContent, 'utf8'); + + // Store diff content in external file + const diffStoragePath = await this.storeDiffToFile(diffId, diffContent); + + const insertDiffQuery = ` + INSERT INTO diff_contents ( + id, commit_id, diff_header, diff_size_bytes, storage_type, + external_storage_path, file_path, change_type, processing_status + ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + ON CONFLICT (commit_id, file_path) + DO UPDATE SET + diff_header = EXCLUDED.diff_header, + diff_size_bytes = EXCLUDED.diff_size_bytes, + storage_type = EXCLUDED.storage_type, + external_storage_path = EXCLUDED.external_storage_path, + change_type = EXCLUDED.change_type, + processing_status = EXCLUDED.processing_status, + updated_at = NOW() + RETURNING id + `; + + const diffResult = await database.query(insertDiffQuery, [ + diffId, + commitId, + `diff --git commit ${commitSha}`, + diffSize, + 'external', + diffStoragePath, + 'multiple_files', + 'modified', + 'pending' + ]); + + // Process file changes if provided + if (fileChanges && fileChanges.length > 0) { + await this.processFileChanges(commitId, fileChanges); + } + + return diffId; + } catch (error) { + console.error('Error storing diff content:', error); + throw error; + } + } + + // Store diff content to external file + async storeDiffToFile(diffId, diffContent) { + console.log('🔍 storeDiffToFile called with:', { + diffId, + hasContent: !!diffContent, + contentLength: diffContent ? diffContent.length : 'undefined' + }); + + if (!diffContent) { + throw new Error('diffContent is undefined in storeDiffToFile'); + } + + // Use DIFF_STORAGE_DIR instead of ATTACHED_REPOS_DIR + const diffDir = process.env.DIFF_STORAGE_DIR || '/home/tech4biz/Desktop/today work/git-diff'; + if (!fs.existsSync(diffDir)) { + fs.mkdirSync(diffDir, { recursive: true }); + } + + const diffFilePath = path.join(diffDir, `${diffId}.diff`); + fs.writeFileSync(diffFilePath, diffContent, 'utf8'); + + return diffFilePath; + } + + // Process file changes for commit + async processFileChanges(commitId, fileChanges) { + for (const fileChange of fileChanges) { + try { + const insertFileQuery = ` + INSERT INTO repository_commit_files ( + commit_id, file_path, change_type, created_at + ) VALUES ($1, $2, $3, NOW()) + `; + + await database.query(insertFileQuery, [ + commitId, + fileChange.file_path, + fileChange.change_type || 'modified' + ]); + } catch (error) { + console.warn(`Error processing file change for ${fileChange.file_path}:`, error.message); + } + } + } + + // Get repository diffs + async getRepositoryDiffs(repositoryId, options = {}) { + const { limit = 10, offset = 0 } = options; + + try { + const query = ` + SELECT + dc.id as diff_id, + dc.diff_header, + dc.diff_size_bytes, + dc.external_storage_path, + dc.change_type, + dc.processing_status, + dc.created_at, + rcd.commit_sha, + rcd.parent_sha, + rcd.message, + rcd.author_name, + rcd.author_email, + rcd.committed_at + FROM diff_contents dc + JOIN repository_commit_details rcd ON dc.commit_id = rcd.id + WHERE rcd.repository_id = $1 + ORDER BY dc.created_at DESC + LIMIT $2 OFFSET $3 + `; + + const result = await database.query(query, [repositoryId, limit, offset]); + return result.rows; + } catch (error) { + console.error('Error getting repository diffs:', error); + throw error; + } + } + + // Analyze diff content + async analyzeDiffContent(repositoryId, diffId, options = {}) { + const { includeContext = true } = options; + + try { + // Get diff content + const diffQuery = ` + SELECT + dc.*, + rcd.commit_sha, + rcd.parent_sha, + rcd.message, + rcd.author_name, + rcd.author_email, + rcd.committed_at + FROM diff_contents dc + JOIN repository_commit_details rcd ON dc.commit_id = rcd.id + WHERE dc.id = $1 AND rcd.repository_id = $2 + `; + + const diffResult = await database.query(diffQuery, [diffId, repositoryId]); + + if (diffResult.rows.length === 0) { + throw new Error('Diff not found'); + } + + const diff = diffResult.rows[0]; + + // Read diff content from file + const diffContent = fs.readFileSync(diff.external_storage_path, 'utf8'); + + // Parse diff content + const parsedDiff = this.parseDiffContent(diffContent); + + // Get repository context if requested + let repositoryContext = null; + if (includeContext) { + repositoryContext = await this.getRepositoryInfo(repositoryId); + } + + // Generate analysis + const analysis = { + diff_id: diffId, + repository_id: repositoryId, + commit_info: { + commit_sha: diff.commit_sha, + parent_sha: diff.parent_sha, + commit_message: diff.message, + author_name: diff.author_name, + author_email: diff.author_email, + commit_date: diff.committed_at + }, + diff_summary: { + total_files_changed: parsedDiff.files.length, + total_additions: parsedDiff.totalAdditions, + total_deletions: parsedDiff.totalDeletions, + files_changed: parsedDiff.files.map(f => ({ + file_path: f.filePath, + change_type: f.changeType, + additions: f.additions, + deletions: f.deletions + })) + }, + analysis_hints: this.generateDiffAnalysisHints(parsedDiff), + repository_context: repositoryContext, + created_at: diff.created_at + }; + + return analysis; + } catch (error) { + console.error('Error analyzing diff content:', error); + throw error; + } + } + + // Parse diff content + parseDiffContent(diffContent) { + console.log('🔍 parseDiffContent called with:', { + hasContent: !!diffContent, + contentType: typeof diffContent, + contentLength: diffContent ? diffContent.length : 'undefined' + }); + + if (!diffContent) { + console.warn('⚠️ parseDiffContent received undefined/null diffContent'); + return { + files: [], + totalAdditions: 0, + totalDeletions: 0 + }; + } + + const files = []; + let totalAdditions = 0; + let totalDeletions = 0; + + const lines = diffContent.split('\n'); + let currentFile = null; + + for (const line of lines) { + if (line.startsWith('diff --git')) { + if (currentFile) { + files.push(currentFile); + } + currentFile = { + filePath: this.extractFilePathFromDiffHeader(line), + changeType: 'modified', + additions: 0, + deletions: 0, + hunks: [] + }; + } else if (line.startsWith('+++') || line.startsWith('---')) { + // Skip file headers + continue; + } else if (line.startsWith('@@')) { + // Hunk header + if (currentFile) { + currentFile.hunks.push({ + header: line, + lines: [] + }); + } + } else if (line.startsWith('+')) { + if (currentFile) { + currentFile.additions++; + totalAdditions++; + } + } else if (line.startsWith('-')) { + if (currentFile) { + currentFile.deletions++; + totalDeletions++; + } + } + } + + if (currentFile) { + files.push(currentFile); + } + + return { + files, + totalAdditions, + totalDeletions + }; + } + + // Extract file path from diff header + extractFilePathFromDiffHeader(header) { + const match = header.match(/diff --git a\/(.+) b\/(.+)/); + if (match) { + return match[2]; // Return the 'b' file path (new version) + } + return 'unknown'; + } + + // Generate diff analysis hints + generateDiffAnalysisHints(parsedDiff) { + const hints = { + complexity_level: 'low', + change_types: [], + affected_areas: [], + risk_level: 'low' + }; + + // Analyze change complexity + const totalChanges = parsedDiff.totalAdditions + parsedDiff.totalDeletions; + if (totalChanges > 1000) { + hints.complexity_level = 'high'; + } else if (totalChanges > 100) { + hints.complexity_level = 'medium'; + } + + // Analyze change types + const changeTypes = new Set(); + const affectedAreas = new Set(); + + for (const file of parsedDiff.files) { + changeTypes.add(file.changeType); + + // Categorize by file type + if (file.filePath.endsWith('.js') || file.filePath.endsWith('.ts')) { + affectedAreas.add('frontend'); + } else if (file.filePath.endsWith('.py')) { + affectedAreas.add('backend'); + } else if (file.filePath.endsWith('.sql')) { + affectedAreas.add('database'); + } else if (file.filePath.endsWith('.md')) { + affectedAreas.add('documentation'); + } + } + + hints.change_types = Array.from(changeTypes); + hints.affected_areas = Array.from(affectedAreas); + + // Assess risk level + if (hints.affected_areas.includes('database') || hints.complexity_level === 'high') { + hints.risk_level = 'high'; + } else if (hints.complexity_level === 'medium' || (parsedDiff.files && parsedDiff.files.length > 10)) { + hints.risk_level = 'medium'; + } + + return hints; + } + + // Create diff streaming session + createDiffStreamingSession(repositoryId, options = {}) { + const sessionId = `diff_stream_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`; + + this.activeStreams.set(sessionId, { + repositoryId, + options, + status: 'initializing', + currentChunk: 0, + totalChunks: 0, + totalFiles: 0, + processedFiles: 0, + startTime: Date.now(), + lastActivity: Date.now(), + sessionType: 'diff_analysis' + }); + + return sessionId; + } + + // Stream diff analysis + async streamDiffAnalysis(sessionId, repositoryId, diffId, options = {}) { + const { includeContext = true } = options; + const startTime = Date.now(); + const chunks = []; + + try { + // Get diff analysis + const analysis = await this.analyzeDiffContent(repositoryId, diffId, { includeContext }); + + // Split analysis into chunks for streaming + const chunkSize = 5; // Files per chunk + const files = analysis.diff_summary.files_changed || []; + const totalChunks = Math.ceil(files.length / chunkSize); + + for (let i = 0; i < files.length; i += chunkSize) { + const chunkFiles = files.slice(i, i + chunkSize); + const chunkNumber = Math.floor(i / chunkSize) + 1; + + const chunk = { + chunk_number: chunkNumber, + total_chunks: totalChunks, + files: chunkFiles, + analysis_hints: analysis.analysis_hints, + progress: { + current_chunk: chunkNumber, + total_chunks: totalChunks, + processed_files: Math.min(i + chunkSize, files.length), + total_files: files.length, + percentage: Math.round((Math.min(i + chunkSize, files.length) / files.length) * 100) + } + }; + + chunks.push(chunk); + } + + const processingTime = Date.now() - startTime; + + return { + chunks, + processingTime, + totalFiles: files.length, + totalChunks + }; + } catch (error) { + console.error('Error streaming diff analysis:', error); + throw error; + } + } + + // Check if this is first-time analysis for a repository + async isFirstTimeAnalysis(repositoryId, commitId) { + try { + const query = ` + SELECT COUNT(*) as count + FROM diff_contents dc + JOIN repository_commit_details rcd ON dc.commit_id = rcd.id + WHERE rcd.repository_id = $1 + `; + + const result = await database.query(query, [repositoryId]); + return parseInt(result.rows[0].count) === 0; + } catch (error) { + console.error('Error checking first-time analysis:', error); + return true; // Default to first-time if error + } + } + + // Get diff content for a specific commit + async getDiffForCommit(repoPath, commitId, isFirstTime) { + try { + // First, try to get stored diff content from database + const storedDiffContent = await this.getStoredDiffFromDatabase(commitId); + if (storedDiffContent) { + console.log(`📁 Using stored diff content from database for commit ${commitId}`); + return storedDiffContent; + } + + // If no stored content in database, check if diff file exists locally + const localDiffContent = await this.getStoredDiffContent(commitId); + if (localDiffContent) { + console.log(`📁 Using local diff file for commit ${commitId}`); + return localDiffContent; + } + + // If no diff content found anywhere, return null (don't generate new) + console.log(`⚠️ No diff content found for commit ${commitId} - skipping analysis`); + return null; + } catch (error) { + console.error('Error getting diff for commit:', error); + return null; + } + } + + // Get stored diff content from database + async getStoredDiffFromDatabase(commitId) { + try { + const query = ` + SELECT dc.external_storage_path, dc.diff_size_bytes + FROM diff_contents dc + JOIN repository_commit_details rcd ON dc.commit_id = rcd.id + WHERE rcd.commit_sha = $1 + `; + + const result = await database.query(query, [commitId]); + + if (result.rows.length > 0) { + const diffRecord = result.rows[0]; + const diffFilePath = diffRecord.external_storage_path; + + if (fs.existsSync(diffFilePath)) { + const content = fs.readFileSync(diffFilePath, 'utf8'); + console.log(`📖 Found diff content from database: ${diffFilePath} (${content ? content.length : 'undefined'} characters)`); + return content; + } + } + + return null; + } catch (error) { + console.error('Error reading diff content from database:', error); + return null; + } + } + + // Get stored diff content from local files + async getStoredDiffContent(commitId) { + try { + const diffDir = process.env.DIFF_STORAGE_DIR || '/home/tech4biz/Desktop/today work/git-diff'; + const commitDir = path.join(diffDir, commitId); + + if (fs.existsSync(commitDir)) { + console.log(`📁 Found commit directory: ${commitDir}`); + + // Read all diff files in the commit directory + const files = fs.readdirSync(commitDir); + const diffFiles = files.filter(file => file.endsWith('.diff')); + + if (diffFiles.length > 0) { + console.log(`📄 Found ${diffFiles.length} diff files in commit directory`); + + // Combine all diff files into one content + let combinedContent = ''; + for (const diffFile of diffFiles) { + const filePath = path.join(commitDir, diffFile); + const content = fs.readFileSync(filePath, 'utf8'); + combinedContent += `\n--- ${diffFile} ---\n${content}\n`; + } + + console.log(`📖 Combined diff content: ${combinedContent ? combinedContent.length : 'undefined'} characters`); + return combinedContent; + } else { + console.log(`❌ No .diff files found in commit directory`); + return null; + } + } + + // Fallback: check for single diff file (old format) + const diffFilePath = path.join(diffDir, `${commitId}.diff`); + if (fs.existsSync(diffFilePath)) { + const content = fs.readFileSync(diffFilePath, 'utf8'); + console.log(`📖 Found stored diff file: ${diffFilePath} (${content ? content.length : 'undefined'} characters)`); + return content; + } + + return null; + } catch (error) { + console.error('Error reading stored diff content:', error); + return null; + } + } + + // Store diff content locally for future use + async storeDiffContentLocally(commitId, diffContent) { + try { + const diffDir = process.env.DIFF_STORAGE_DIR || '/home/tech4biz/Desktop/today work/git-diff'; + if (!fs.existsSync(diffDir)) { + fs.mkdirSync(diffDir, { recursive: true }); + } + + const diffFilePath = path.join(diffDir, `${commitId}.diff`); + fs.writeFileSync(diffFilePath, diffContent, 'utf8'); + console.log(`💾 Stored diff content locally: ${diffFilePath}`); + + return diffFilePath; + } catch (error) { + console.error('Error storing diff content locally:', error); + return null; + } + } + + // Generate rich diff metadata using existing database tables + async generateRichDiffMetadata(options) { + const { + repositoryId, + commitId, + diffContent, + repositoryName, + ownerName, + isFirstTime, + includeContext + } = options; + + console.log('🔍 generateRichDiffMetadata called with:', { + repositoryId, + commitId, + hasDiffContent: !!diffContent, + diffContentLength: diffContent ? diffContent.length : 'undefined', + repositoryName, + ownerName, + isFirstTime + }); + + // Get commit details from database + const commitDetails = await this.getCommitDetailsFromDatabase(commitId); + + // Get file changes from database + const fileChanges = await this.getFileChangesFromDatabase(commitId); + + // Parse diff content for additional analysis + console.log('🔍 Parsing diff content...'); + const parsedDiff = this.parseDiffContent(diffContent); + console.log('📊 Parsed diff result:', { + hasFiles: !!parsedDiff.files, + filesLength: parsedDiff.files ? parsedDiff.files.length : 'undefined', + totalAdditions: parsedDiff.totalAdditions, + totalDeletions: parsedDiff.totalDeletions + }); + + // Ensure parsedDiff has the expected structure + if (!parsedDiff || !parsedDiff.files) { + console.warn('⚠️ Parsed diff is missing files array, using empty array'); + parsedDiff.files = []; + } + if (!parsedDiff.totalAdditions) parsedDiff.totalAdditions = 0; + if (!parsedDiff.totalDeletions) parsedDiff.totalDeletions = 0; + + // Generate comprehensive metadata using database data + const metadata = { + repository_id: repositoryId, + commit_id: commitId, + repository_name: repositoryName, + owner_name: ownerName, + full_name: `${ownerName}/${repositoryName}`, + is_first_time: isFirstTime, + analysis_timestamp: new Date().toISOString(), + + // Commit details from database + commit_details: commitDetails, + + // File changes from database + file_changes: fileChanges, + + // Additional analysis from diff content + diff_analysis: { + total_additions: parsedDiff.totalAdditions, + total_deletions: parsedDiff.totalDeletions, + total_changes: parsedDiff.totalAdditions + parsedDiff.totalDeletions, + files_changed: parsedDiff.files.map(file => ({ + file_path: file.filePath, + change_type: file.changeType, + additions: file.additions, + deletions: file.deletions, + changes: file.additions + file.deletions, + file_extension: path.extname(file.filePath), + language: this.detectLanguage(path.extname(file.filePath)), + is_binary: this.isBinaryFile(file.filePath), + complexity_score: this.calculateFileComplexity(file) + })) + }, + + // Analysis metadata + complexity_score: this.calculateOverallComplexity(parsedDiff), + risk_level: this.assessRiskLevel(parsedDiff), + affected_areas: this.getAffectedAreas(parsedDiff.files || []), + change_categories: this.categorizeChanges(parsedDiff.files || []), + + // Repository context (if requested) + repository_context: includeContext ? await this.getRepositoryContext(repositoryId) : null, + + // Processing metadata + diff_size_bytes: diffContent ? Buffer.byteLength(diffContent, 'utf8') : 0, + processing_time_ms: Date.now(), + analysis_version: '1.0' + }; + + return metadata; + } + + // Store diff for analysis + async storeDiffForAnalysis(options) { + const { + repositoryId, + commitId, + diffContent, + diffMetadata, + isFirstTime + } = options; + + try { + // Create or get commit record + let commitId_db; + const commitQuery = ` + SELECT id FROM repository_commit_details + WHERE repository_id = $1 AND commit_sha = $2 + `; + const commitResult = await database.query(commitQuery, [repositoryId, commitId]); + + if (commitResult.rows.length === 0) { + // Create new commit record + const insertCommitQuery = ` + INSERT INTO repository_commit_details ( + repository_id, commit_sha, message, + author_name, author_email, committed_at, created_at + ) VALUES ($1, $2, $3, $4, $5, $6, NOW()) + RETURNING id + `; + const commitInsertResult = await database.query(insertCommitQuery, [ + repositoryId, + commitId, + diffMetadata.message || 'Auto-generated commit', + diffMetadata.author_name || 'System', + diffMetadata.author_email || 'system@example.com', + new Date().toISOString() + ]); + commitId_db = commitInsertResult.rows[0].id; + } else { + commitId_db = commitResult.rows[0].id; + } + + // Store diff content + const diffId = require('crypto').randomUUID(); + console.log('🔍 About to calculate diffSize in storeDiffForAnalysis:', { + hasDiffContent: !!diffContent, + diffContentType: typeof diffContent, + diffContentLength: diffContent ? diffContent.length : 'undefined' + }); + const diffSize = Buffer.byteLength(diffContent, 'utf8'); + + // Store diff content in external file + const diffStoragePath = await this.storeDiffToFile(diffId, diffContent); + + const insertDiffQuery = ` + INSERT INTO diff_contents ( + id, commit_id, diff_header, diff_size_bytes, storage_type, + external_storage_path, file_path, change_type, processing_status + ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + ON CONFLICT (commit_id, file_path) + DO UPDATE SET + diff_header = EXCLUDED.diff_header, + diff_size_bytes = EXCLUDED.diff_size_bytes, + storage_type = EXCLUDED.storage_type, + external_storage_path = EXCLUDED.external_storage_path, + change_type = EXCLUDED.change_type, + processing_status = EXCLUDED.processing_status, + updated_at = NOW() + RETURNING id + `; + + await database.query(insertDiffQuery, [ + diffId, + commitId_db, + `diff --git commit ${commitId}`, + diffSize, + 'external', + diffStoragePath, + 'multiple_files', + isFirstTime ? 'initial' : 'incremental', + 'processed' + ]); + + return diffId; + } catch (error) { + console.error('Error storing diff for analysis:', error); + throw error; + } + } + + // Generate full analysis + async generateFullAnalysis(diffMetadata, repositoryId) { + // Get the actual diff content for AI analysis + const actualDiffContent = await this.getStoredDiffContent(diffMetadata.commit_id); + + return { + analysis_type: 'full', + summary: { + description: 'Complete repository analysis with full context', + scope: 'All files and dependencies', + confidence_level: 'high' + }, + insights: { + architecture_impact: this.analyzeArchitectureImpact(diffMetadata), + performance_impact: this.analyzePerformanceImpact(diffMetadata), + security_considerations: this.analyzeSecurityImpact(diffMetadata), + maintainability_score: this.calculateMaintainabilityScore(diffMetadata) + }, + recommendations: this.generateFullRecommendations(diffMetadata), + next_steps: this.generateNextSteps(diffMetadata, 'full'), + // Include actual diff content for AI analysis + diff_content: { + raw_content: actualDiffContent, + content_length: actualDiffContent ? actualDiffContent.length : 0, + content_preview: actualDiffContent ? actualDiffContent.substring(0, 500) + '...' : null, + file_path: actualDiffContent ? `${diffMetadata.commit_id}.diff` : null + } + }; + } + + // Generate incremental analysis + async generateIncrementalAnalysis(diffMetadata, repositoryId, commitId) { + // Get the actual diff content for AI analysis + const actualDiffContent = await this.getStoredDiffContent(commitId); + + return { + analysis_type: 'incremental', + summary: { + description: 'Incremental analysis focusing on changes', + scope: 'Modified files and their dependencies', + confidence_level: 'medium' + }, + insights: { + change_impact: this.analyzeChangeImpact(diffMetadata), + regression_risk: this.assessRegressionRisk(diffMetadata), + integration_points: this.identifyIntegrationPoints(diffMetadata), + testing_requirements: this.suggestTestingRequirements(diffMetadata) + }, + recommendations: this.generateIncrementalRecommendations(diffMetadata), + next_steps: this.generateNextSteps(diffMetadata, 'incremental'), + // Include actual diff content for AI analysis + diff_content: { + raw_content: actualDiffContent, + content_length: actualDiffContent ? actualDiffContent.length : 0, + content_preview: actualDiffContent ? actualDiffContent.substring(0, 500) + '...' : null, + file_path: actualDiffContent ? `${commitId}.diff` : null + } + }; + } + + // Stream analysis chunks + async streamAnalysisChunks(analysis, diffMetadata) { + const chunks = []; + const chunkSize = 3; // Files per chunk + + // Get actual diff content for streaming + const actualDiffContent = await this.getStoredDiffContent(diffMetadata.commit_id); + + const filesChanged = diffMetadata.files_changed || []; + + for (let i = 0; i < filesChanged.length; i += chunkSize) { + const chunkFiles = filesChanged.slice(i, i + chunkSize); + const chunkNumber = Math.floor(i / chunkSize) + 1; + + const chunk = { + chunk_number: chunkNumber, + total_chunks: Math.ceil(filesChanged.length / chunkSize), + files: chunkFiles, + analysis: { + insights: analysis.insights, + recommendations: analysis.recommendations.slice(0, chunkNumber * 2) // Limit recommendations per chunk + }, + // Include diff content in each chunk + diff_content: { + raw_content: actualDiffContent, + content_length: actualDiffContent ? actualDiffContent.length : 0, + content_preview: actualDiffContent ? actualDiffContent.substring(0, 500) + '...' : null, + file_path: actualDiffContent ? `${diffMetadata.commit_id}.diff` : null + }, + progress: { + current_chunk: chunkNumber, + total_chunks: Math.ceil(filesChanged.length / chunkSize), + processed_files: Math.min(i + chunkSize, filesChanged.length), + total_files: filesChanged.length, + percentage: Math.round((Math.min(i + chunkSize, filesChanged.length) / filesChanged.length) * 100) + } + }; + + chunks.push(chunk); + } + + return chunks; + } + + // Helper methods for analysis + calculateFileComplexity(file) { + const baseComplexity = file.additions + file.deletions; + const extension = path.extname(file.filePath); + + // Adjust complexity based on file type + const complexityMultipliers = { + '.js': 1.2, + '.ts': 1.3, + '.py': 1.1, + '.java': 1.4, + '.cpp': 1.5, + '.c': 1.3 + }; + + const multiplier = complexityMultipliers[extension] || 1.0; + return Math.round(baseComplexity * multiplier); + } + + calculateOverallComplexity(parsedDiff) { + if (!parsedDiff) return 'low'; + + const totalChanges = (parsedDiff.totalAdditions || 0) + (parsedDiff.totalDeletions || 0); + const fileCount = parsedDiff.files ? parsedDiff.files.length : 0; + + if (totalChanges > 1000 || fileCount > 20) return 'high'; + if (totalChanges > 100 || fileCount > 5) return 'medium'; + return 'low'; + } + + assessRiskLevel(parsedDiff) { + if (!parsedDiff) return 'low'; + + const totalChanges = (parsedDiff.totalAdditions || 0) + (parsedDiff.totalDeletions || 0); + const fileCount = parsedDiff.files ? parsedDiff.files.length : 0; + + // Check for high-risk file types + const highRiskFiles = parsedDiff.files ? parsedDiff.files.filter(file => + file.filePath.includes('config') || + file.filePath.includes('database') || + file.filePath.includes('security') + ) : []; + + if (highRiskFiles.length > 0 || totalChanges > 500) return 'high'; + if (totalChanges > 100 || fileCount > 10) return 'medium'; + return 'low'; + } + + getAffectedAreas(files) { + const areas = new Set(); + + if (!files || !Array.isArray(files)) { + return []; + } + + files.forEach(file => { + const path = file.filePath.toLowerCase(); + if (path.includes('frontend') || path.includes('ui') || path.includes('component')) { + areas.add('frontend'); + } + if (path.includes('backend') || path.includes('api') || path.includes('service')) { + areas.add('backend'); + } + if (path.includes('database') || path.includes('sql') || path.includes('migration')) { + areas.add('database'); + } + if (path.includes('test') || path.includes('spec')) { + areas.add('testing'); + } + if (path.includes('config') || path.includes('env')) { + areas.add('configuration'); + } + }); + + return Array.from(areas); + } + + categorizeChanges(files) { + if (!files || !Array.isArray(files)) { + return { + new_features: 0, + bug_fixes: 0, + refactoring: 0, + documentation: 0, + configuration: 0 + }; + } + + const categories = { + new_features: files.filter(f => f.changeType === 'added').length, + bug_fixes: files.filter(f => f.changeType === 'modified' && f.filePath.includes('fix')).length, + refactoring: files.filter(f => f.changeType === 'modified' && f.additions === f.deletions).length, + documentation: files.filter(f => f.filePath.endsWith('.md')).length, + configuration: files.filter(f => f.filePath.includes('config')).length + }; + + return categories; + } + + // Additional helper methods would be implemented here... + analyzeArchitectureImpact(diffMetadata) { + return { + impact_level: diffMetadata.complexity_score, + affected_components: diffMetadata.affected_areas, + architectural_changes: (diffMetadata.files_changed || []).filter(f => f.filePath.includes('architecture')).length + }; + } + + analyzePerformanceImpact(diffMetadata) { + return { + potential_impact: diffMetadata.complexity_score, + performance_critical_files: (diffMetadata.files_changed || []).filter(f => + f.filePath.includes('performance') || f.filePath.includes('optimization') + ).length + }; + } + + analyzeSecurityImpact(diffMetadata) { + return { + security_relevant_changes: (diffMetadata.files_changed || []).filter(f => + f.filePath.includes('auth') || f.filePath.includes('security') || f.filePath.includes('crypto') + ).length, + risk_assessment: diffMetadata.risk_level + }; + } + + calculateMaintainabilityScore(diffMetadata) { + const complexity = diffMetadata.complexity_score === 'high' ? 3 : diffMetadata.complexity_score === 'medium' ? 2 : 1; + const fileCount = (diffMetadata.files_changed || []).length; + const risk = diffMetadata.risk_level === 'high' ? 3 : diffMetadata.risk_level === 'medium' ? 2 : 1; + + return Math.max(1, 10 - (complexity + fileCount + risk)); + } + + generateFullRecommendations(diffMetadata) { + const recommendations = []; + + if (diffMetadata.complexity_score === 'high') { + recommendations.push('Consider breaking down complex changes into smaller, manageable commits'); + } + + if (diffMetadata.risk_level === 'high') { + recommendations.push('Implement comprehensive testing before deployment'); + } + + if (diffMetadata.affected_areas.includes('database')) { + recommendations.push('Review database migration scripts and backup procedures'); + } + + return recommendations; + } + + generateIncrementalRecommendations(diffMetadata) { + const recommendations = []; + + recommendations.push('Review changed files for potential side effects'); + recommendations.push('Run relevant test suites for modified components'); + + if (diffMetadata.affected_areas.includes('frontend')) { + recommendations.push('Test UI changes across different browsers and devices'); + } + + if (diffMetadata.affected_areas.includes('backend')) { + recommendations.push('Verify API endpoints and data validation'); + } + + return recommendations; + } + + generateNextSteps(diffMetadata, analysisType) { + const steps = []; + + if (analysisType === 'full') { + steps.push('Review overall architecture and design patterns'); + steps.push('Plan comprehensive testing strategy'); + steps.push('Document architectural decisions'); + } else { + steps.push('Test specific changes in isolation'); + steps.push('Verify integration with existing components'); + steps.push('Update relevant documentation'); + } + + return steps; + } + + analyzeChangeImpact(diffMetadata) { + return { + scope: diffMetadata.affected_areas, + magnitude: diffMetadata.complexity_score, + files_affected: (diffMetadata.files_changed || []).length + }; + } + + assessRegressionRisk(diffMetadata) { + return { + risk_level: diffMetadata.risk_level, + critical_files: (diffMetadata.files_changed || []).filter(f => f.filePath.includes('core')).length, + testing_priority: diffMetadata.complexity_score + }; + } + + identifyIntegrationPoints(diffMetadata) { + return (diffMetadata.files_changed || []).filter(f => + f.filePath.includes('api') || f.filePath.includes('interface') || f.filePath.includes('service') + ).map(f => f.filePath); + } + + suggestTestingRequirements(diffMetadata) { + const requirements = ['Unit tests for modified functions']; + + if (diffMetadata.affected_areas.includes('frontend')) { + requirements.push('UI component testing'); + } + + if (diffMetadata.affected_areas.includes('backend')) { + requirements.push('API endpoint testing'); + } + + if (diffMetadata.affected_areas.includes('database')) { + requirements.push('Database integration testing'); + } + + return requirements; + } + + async getRepositoryContext(repositoryId) { + try { + return await this.getRepositoryInfo(repositoryId); + } catch (error) { + console.error('Error getting repository context:', error); + return null; + } + } + + // Get commit details from database + async getCommitDetailsFromDatabase(commitId) { + try { + const query = ` + SELECT + commit_sha, + message, + author_name, + author_email, + committed_at, + created_at + FROM repository_commit_details + WHERE commit_sha = $1 + `; + + const result = await database.query(query, [commitId]); + + if (result.rows.length > 0) { + return result.rows[0]; + } + + return null; + } catch (error) { + console.error('Error getting commit details from database:', error); + return null; + } + } + + // Get file changes from database + async getFileChangesFromDatabase(commitId) { + try { + const query = ` + SELECT + rcf.id, + rcf.commit_id, + rcf.file_path, + rcf.change_type, + rcf.created_at + FROM repository_commit_files rcf + JOIN repository_commit_details rcd ON rcf.commit_id = rcd.id + WHERE rcd.commit_sha = $1 + ORDER BY rcf.file_path + `; + + const result = await database.query(query, [commitId]); + return result.rows; + } catch (error) { + console.error('Error getting file changes from database:', error); + return []; + } + } + + // Get simple file changes + async getSimpleFileChanges(commitId) { + try { + // First try repository_commit_files table + const commitQuery = ` + SELECT id FROM repository_commit_details + WHERE commit_sha = $1 + `; + + const commitResult = await database.query(commitQuery, [commitId]); + + if (commitResult.rows.length > 0) { + const internalCommitId = commitResult.rows[0].id; + + const fileQuery = ` + SELECT + rcf.file_path, + rcf.change_type + FROM repository_commit_files rcf + WHERE rcf.commit_id = $1 + `; + + const result = await database.query(fileQuery, [internalCommitId]); + + if (result.rows.length > 0) { + return result.rows; + } + } + + // If no data in repository_commit_files, try diff_contents table + const diffQuery = ` + SELECT + dc.file_path, + dc.change_type + FROM diff_contents dc + WHERE dc.commit_id = $1 + `; + + const diffResult = await database.query(diffQuery, [commitId]); + return diffResult.rows; + } catch (error) { + console.error('Error getting file changes:', error); + return []; + } + } + + // Get actual diff content from database first, then local files + async getActualDiffContent(commitId) { + try { + // First check database - query directly by commit_id + const query = ` + SELECT + dc.external_storage_path, + dc.file_path, + dc.change_type + FROM diff_contents dc + WHERE dc.commit_id = $1 + LIMIT 1 + `; + + const result = await database.query(query, [commitId]); + + if (result.rows.length === 0) { + return null; // No data in database + } + + const diffInfo = result.rows[0]; + const fs = require('fs'); + const path = require('path'); + + // Read the actual diff file + if (fs.existsSync(diffInfo.external_storage_path)) { + const diffContent = fs.readFileSync(diffInfo.external_storage_path, 'utf8'); + return { + file_path: diffInfo.file_path, + change_type: diffInfo.change_type, + local_path: diffInfo.external_storage_path, + content: diffContent, + size: diffContent.length + }; + } + + return null; + } catch (error) { + console.error('Error getting diff content:', error); + return null; + } + } + + // ==================== BULK COMMIT ANALYSIS METHODS ==================== + + // Get multiple commits with their diff contents + async getBulkCommitDetails(commitIds) { + try { + console.log(`🔍 Getting bulk commit details for ${commitIds.length} commits`); + + const results = []; + + for (const commitId of commitIds) { + try { + // Get commit details + const commitQuery = ` + SELECT + id, + commit_sha, + message, + committed_at + FROM repository_commit_details + WHERE id = $1 + `; + + const commitResult = await database.query(commitQuery, [commitId]); + + if (commitResult.rows.length === 0) { + results.push({ + commitId: commitId, + status: 'not_found', + message: 'Commit not found in database' + }); + continue; + } + + const commitDetails = commitResult.rows[0]; + + // Get file changes for this commit + const fileChanges = await this.getSimpleFileChanges(commitId); + + // Get diff content for this commit + const diffContent = await this.getActualDiffContent(commitId); + + results.push({ + commitId: commitId, + status: 'success', + commitDetails: commitDetails, + fileChanges: fileChanges, + diffContent: diffContent, + filesCount: fileChanges.length, + hasDiffContent: !!diffContent + }); + + } catch (error) { + console.error(`Error processing commit ${commitId}:`, error); + results.push({ + commitId: commitId, + status: 'error', + message: error.message + }); + } + } + + return results; + } catch (error) { + console.error('Error in getBulkCommitDetails:', error); + throw error; + } + } + + // Batch file reading for multiple commits + async batchReadDiffFiles(commitResults) { + try { + console.log(`📁 Batch reading diff files for ${commitResults.length} commits`); + + const fileReadPromises = []; + + for (const commitResult of commitResults) { + if (commitResult.status === 'success' && commitResult.diffContent) { + const filePath = commitResult.diffContent.local_path; + + if (filePath) { + fileReadPromises.push( + this.readSingleDiffFile(filePath, commitResult.commitId) + ); + } + } + } + + // Read all files in parallel + const fileContents = await Promise.all(fileReadPromises); + + // Merge results back with commit data + const enrichedResults = commitResults.map(commitResult => { + if (commitResult.status === 'success') { + const fileContent = fileContents.find(fc => fc.commitId === commitResult.commitId); + if (fileContent) { + commitResult.diffContent.content = fileContent.content; + commitResult.diffContent.size = fileContent.size; + commitResult.diffContent.readStatus = fileContent.status; + } + } + return commitResult; + }); + + return enrichedResults; + } catch (error) { + console.error('Error in batchReadDiffFiles:', error); + throw error; + } + } + + // Read a single diff file + async readSingleDiffFile(filePath, commitId) { + try { + const fs = require('fs'); + + if (fs.existsSync(filePath)) { + const content = fs.readFileSync(filePath, 'utf8'); + return { + commitId: commitId, + filePath: filePath, + content: content, + size: content.length, + status: 'success' + }; + } else { + return { + commitId: commitId, + filePath: filePath, + content: null, + size: 0, + status: 'file_not_found' + }; + } + } catch (error) { + console.error(`Error reading file ${filePath}:`, error); + return { + commitId: commitId, + filePath: filePath, + content: null, + size: 0, + status: 'error', + error: error.message + }; + } + } + + // Get bulk analysis summary + async getBulkAnalysisSummary(commitResults) { + try { + const summary = { + total_commits: commitResults.length, + successful_commits: commitResults.filter(r => r.status === 'success').length, + failed_commits: commitResults.filter(r => r.status === 'error').length, + not_found_commits: commitResults.filter(r => r.status === 'not_found').length, + total_files: commitResults.reduce((sum, r) => sum + (r.filesCount || 0), 0), + commits_with_diff: commitResults.filter(r => r.hasDiffContent).length, + total_content_size: commitResults.reduce((sum, r) => { + return sum + (r.diffContent?.size || 0); + }, 0) + }; + + return summary; + } catch (error) { + console.error('Error in getBulkAnalysisSummary:', error); + throw error; + } + } + + // Process bulk commits for AI analysis + async processBulkCommitsForAI(commitResults) { + try { + console.log(`🤖 Processing ${commitResults.length} commits for AI analysis`); + + const aiInputs = []; + + for (const commitResult of commitResults) { + if (commitResult.status === 'success' && commitResult.diffContent?.content) { + aiInputs.push({ + commitId: commitResult.commitId, + commitDetails: commitResult.commitDetails, + fileChanges: commitResult.fileChanges, + diffContent: commitResult.diffContent, + analysisReady: true + }); + } + } + + return aiInputs; + } catch (error) { + console.error('Error in processBulkCommitsForAI:', error); + throw error; + } + } + + // Helper method to detect if file is binary + isBinaryFile(filePath) { + const binaryExtensions = ['.exe', '.dll', '.so', '.dylib', '.bin', '.img', '.iso', '.zip', '.tar', '.gz', '.rar', '.7z', '.pdf', '.doc', '.docx', '.xls', '.xlsx', '.ppt', '.pptx', '.jpg', '.jpeg', '.png', '.gif', '.bmp', '.tiff', '.svg', '.ico', '.woff', '.woff2', '.ttf', '.eot']; + const extension = path.extname(filePath).toLowerCase(); + return binaryExtensions.includes(extension); + } + + // Helper method to detect language from file extension + detectLanguage(fileExtension) { + const languageMap = { + '.js': 'javascript', + '.jsx': 'javascript', + '.ts': 'typescript', + '.tsx': 'typescript', + '.py': 'python', + '.java': 'java', + '.cpp': 'cpp', + '.c': 'c', + '.cs': 'csharp', + '.php': 'php', + '.rb': 'ruby', + '.go': 'go', + '.rs': 'rust', + '.kt': 'kotlin', + '.swift': 'swift', + '.scala': 'scala', + '.clj': 'clojure', + '.hs': 'haskell', + '.elm': 'elm', + '.ml': 'ocaml', + '.fs': 'fsharp', + '.vb': 'vbnet', + '.html': 'html', + '.css': 'css', + '.scss': 'scss', + '.json': 'json', + '.yaml': 'yaml', + '.yml': 'yaml', + '.xml': 'xml', + '.sql': 'sql', + '.sh': 'bash', + '.md': 'markdown' + }; + + return languageMap[fileExtension] || 'unknown'; + } +} + +module.exports = AIStreamingService; diff --git a/services/git-integration/src/routes/github-integration.routes.js b/services/git-integration/src/routes/github-integration.routes.js index 53ba845..8b2a1c5 100644 --- a/services/git-integration/src/routes/github-integration.routes.js +++ b/services/git-integration/src/routes/github-integration.routes.js @@ -921,6 +921,85 @@ router.get('/repository/:id/files', async (req, res) => { } }); +// Resolve repository path (case-insensitive path resolution) +router.get('/repository/:id/resolve-path', async (req, res) => { + try { + const { id } = req.params; + const { file_path } = req.query; + + if (!file_path) { + return res.status(400).json({ + success: false, + message: 'file_path query parameter is required' + }); + } + + // Get repository storage path + const storageQ = `SELECT local_path FROM repository_storage WHERE repository_id = $1 ORDER BY created_at DESC LIMIT 1`; + const storageRes = await database.query(storageQ, [id]); + + if (storageRes.rows.length === 0) { + return res.status(404).json({ + success: false, + message: 'Repository not stored locally' + }); + } + + const localBase = storageRes.rows[0].local_path; + const pathJoin = require('path').join; + const fs = require('fs'); + + // Helper: case-insensitive resolution + const resolveCaseInsensitive = (base, rel) => { + const parts = rel.split('/').filter(Boolean); + let cur = base; + for (const p of parts) { + if (!fs.existsSync(cur)) return null; + const entries = fs.readdirSync(cur); + const match = entries.find(e => e.toLowerCase() === p.toLowerCase()); + if (!match) return null; + cur = pathJoin(cur, match); + } + return cur; + }; + + let absPath = pathJoin(localBase, file_path); + let exists = fs.existsSync(absPath); + let isDirectory = false; + + if (!exists) { + absPath = resolveCaseInsensitive(localBase, file_path); + if (absPath) { + exists = fs.existsSync(absPath); + if (exists) { + isDirectory = fs.statSync(absPath).isDirectory(); + } + } + } else { + isDirectory = fs.statSync(absPath).isDirectory(); + } + + res.json({ + success: true, + data: { + repository_id: id, + local_path: localBase, + requested_file_path: file_path, + resolved_absolute_path: absPath, + exists: exists, + is_directory: isDirectory + } + }); + + } catch (error) { + console.error('Error resolving repository path:', error); + res.status(500).json({ + success: false, + message: error.message || 'Failed to resolve repository path' + }); + } +}); + // Get file content router.get('/repository/:id/file-content', async (req, res) => { try { diff --git a/services/requirement-processor/Dockerfile b/services/requirement-processor/Dockerfile index 1f7cfac..efa702b 100644 --- a/services/requirement-processor/Dockerfile +++ b/services/requirement-processor/Dockerfile @@ -9,7 +9,7 @@ RUN apt-get update && apt-get install -y \ # Copy requirements and install Python dependencies COPY requirements.txt . -RUN pip install --no-cache-dir -r requirements.txt +RUN pip install -r requirements.txt # Copy application code COPY src/ ./src/ diff --git a/test-microservices-architecture.js b/test-microservices-architecture.js new file mode 100644 index 0000000..57884cd --- /dev/null +++ b/test-microservices-architecture.js @@ -0,0 +1,150 @@ +#!/usr/bin/env node + +/** + * Microservices Architecture Test Script + * Tests: Frontend (3001) → API Gateway (8000) → Git Integration (8012) + */ + +const axios = require('axios'); + +const SERVICES = { + FRONTEND: 'http://localhost:3001', + API_GATEWAY: 'http://localhost:8000', + GIT_INTEGRATION: 'http://localhost:8012' +}; + +const TEST_ENDPOINTS = [ + // GitHub Integration Endpoints + { path: '/api/github/health', method: 'GET', service: 'git-integration' }, + { path: '/api/github/auth/github/status', method: 'GET', service: 'git-integration' }, + + // Diff Viewer Endpoints + { path: '/api/diffs/repositories', method: 'GET', service: 'git-integration' }, + + // AI Streaming Endpoints + { path: '/api/ai/streaming-sessions', method: 'GET', service: 'git-integration' }, + + // VCS Endpoints + { path: '/api/vcs/github/auth/start', method: 'GET', service: 'git-integration' } +]; + +async function testServiceHealth(serviceName, baseUrl) { + console.log(`\n🔍 Testing ${serviceName} health...`); + try { + const response = await axios.get(`${baseUrl}/health`, { timeout: 5000 }); + console.log(`✅ ${serviceName} is healthy: ${response.status}`); + return true; + } catch (error) { + console.log(`❌ ${serviceName} is unhealthy: ${error.message}`); + return false; + } +} + +async function testEndpoint(endpoint) { + console.log(`\n🔍 Testing ${endpoint.method} ${endpoint.path}...`); + + // Test direct service call + try { + const directUrl = `${SERVICES.GIT_INTEGRATION}${endpoint.path}`; + const directResponse = await axios({ + method: endpoint.method, + url: directUrl, + timeout: 10000, + validateStatus: () => true + }); + console.log(`✅ Direct service call: ${directResponse.status}`); + } catch (error) { + console.log(`❌ Direct service call failed: ${error.message}`); + } + + // Test through API Gateway + try { + const gatewayUrl = `${SERVICES.API_GATEWAY}${endpoint.path}`; + const gatewayResponse = await axios({ + method: endpoint.method, + url: gatewayUrl, + timeout: 10000, + validateStatus: () => true + }); + console.log(`✅ Gateway call: ${gatewayResponse.status}`); + } catch (error) { + console.log(`❌ Gateway call failed: ${error.message}`); + } +} + +async function testFrontendIntegration() { + console.log(`\n🔍 Testing Frontend Integration...`); + + // Test if frontend is running + try { + const response = await axios.get(SERVICES.FRONTEND, { timeout: 5000 }); + console.log(`✅ Frontend is accessible: ${response.status}`); + } catch (error) { + console.log(`❌ Frontend is not accessible: ${error.message}`); + return false; + } + + return true; +} + +async function runArchitectureTest() { + console.log('🚀 Starting Microservices Architecture Test'); + console.log('='.repeat(60)); + + // Test service health + const gitIntegrationHealthy = await testServiceHealth('Git Integration Service', SERVICES.GIT_INTEGRATION); + const apiGatewayHealthy = await testServiceHealth('API Gateway', SERVICES.API_GATEWAY); + const frontendHealthy = await testFrontendIntegration(); + + console.log('\n📊 Service Health Summary:'); + console.log(`Git Integration (8012): ${gitIntegrationHealthy ? '✅' : '❌'}`); + console.log(`API Gateway (8000): ${apiGatewayHealthy ? '✅' : '❌'}`); + console.log(`Frontend (3001): ${frontendHealthy ? '✅' : '❌'}`); + + if (!gitIntegrationHealthy || !apiGatewayHealthy) { + console.log('\n❌ Critical services are down. Please start the services first.'); + console.log('Run: docker-compose up -d'); + return; + } + + // Test endpoints + console.log('\n🔍 Testing Endpoints...'); + for (const endpoint of TEST_ENDPOINTS) { + await testEndpoint(endpoint); + } + + // Test routing flow + console.log('\n🔍 Testing Routing Flow...'); + console.log('Frontend (3001) → API Gateway (8000) → Git Integration (8012)'); + + // Test a simple endpoint through the complete flow + try { + const testUrl = `${SERVICES.API_GATEWAY}/api/github/health`; + const response = await axios.get(testUrl, { timeout: 10000 }); + console.log(`✅ Complete flow test: ${response.status}`); + console.log(`Response: ${JSON.stringify(response.data, null, 2)}`); + } catch (error) { + console.log(`❌ Complete flow test failed: ${error.message}`); + } + + console.log('\n🎯 Architecture Test Complete'); + console.log('='.repeat(60)); + + if (gitIntegrationHealthy && apiGatewayHealthy && frontendHealthy) { + console.log('✅ All services are healthy and properly configured!'); + console.log('\n📋 Next Steps:'); + console.log('1. Start all services: docker-compose up -d'); + console.log('2. Test frontend at: http://localhost:3001'); + console.log('3. Test API Gateway at: http://localhost:8000'); + console.log('4. Test Git Integration at: http://localhost:8012'); + } else { + console.log('❌ Some services need attention. Check the logs above.'); + } +} + +// Run the test +if (require.main === module) { + runArchitectureTest().catch(console.error); +} + +module.exports = { runArchitectureTest, testServiceHealth, testEndpoint };