codenuk_backend_mine/services/git-integration/FLOW_DOCUMENTATION.md

Git Integration Service - Complete Flow Documentation

Overview

This document describes the complete flow of the git-integration service for attaching GitHub repositories.

Configuration

Environment Variables

• GITHUB_CLIENT_ID: OAuth App Client ID
• GITHUB_CLIENT_SECRET: OAuth App Client Secret
• GITHUB_REDIRECT_URI: OAuth callback URL (http://localhost:8000/api/github/auth/github/callback)
• GITHUB_WEBHOOK_SECRET: mywebhooksecret2025
• PUBLIC_BASE_URL: https://98fa7f21e2e9.ngrok-free.app (your ngrok URL)
• ATTACHED_REPOS_DIR: /app/git-repos
• DIFF_STORAGE_DIR: /app/git-repos/diffs
• FRONTEND_URL: http://localhost:3001

Storage Location

• Container Path: /app/git-repos
• Host Path: /home/tech4biz/Desktop/today work/git-repo
• Repository Format: owner__REPONAME__branch/

Complete User Flow

1. User Attaches Repository

Endpoint: POST /api/github/attach-repository

Request Body:

{
  "repository_url": "https://github.com/owner/repo",
  "branch_name": "main"
}

Headers:

• x-user-id: User's unique ID

2. Public/Private Detection

The service automatically detects if the repository is public or private:

For Public Repositories:

1. ✅ Attempts unauthenticated access via GitHub API
2. ✅ If successful, proceeds to clone without OAuth
3. ✅ Clones repository using git clone (full clone with .git directory)
4. ✅ Stores all files including .git folder
5. ✅ Creates webhook automatically (if PUBLIC_BASE_URL is set)

For Private Repositories:

1. ❌ Unauthenticated access fails (404 error)
2. 🔐 Checks if user has GitHub OAuth token
3. If NOT authenticated:
   • Returns 401 with OAuth URL
   • User is redirected to GitHub for authentication
4. If authenticated:
   • Uses OAuth token to access repository
   • Proceeds to clone with authentication

3. GitHub OAuth Flow (Private Repos Only)

Step 1: Initiate OAuth

• Frontend receives auth_url from backend
• User clicks "Connect GitHub" button
• Redirects to: http://localhost:8000/api/github/auth/github?redirect=1&user_id={userId}&state={state}

Step 2: GitHub Authorization

• User authorizes the app on GitHub
• GitHub redirects to callback URL with authorization code

Step 3: OAuth Callback

• Endpoint: GET /api/github/auth/github/callback
• Exchanges code for access token
• Stores token in database (table: github_user_tokens)
• Auto-attaches repository if state contains repo context
• Redirects back to frontend: {FRONTEND_URL}/project-builder?github_connected=1&repo_attached=1

4. Repository Cloning

Clone Method: Full git clone (includes .git directory)

For Public Repos:

git clone -b {branch} https://github.com/{owner}/{repo}.git

For Private Repos:

git clone -b {branch} https://oauth2:{token}@github.com/{owner}/{repo}.git

Storage Structure:

/home/tech4biz/Desktop/today work/git-repo/
├── owner__REPONAME__branch/
│   ├── .git/              # ✅ Full git history included
│   ├── src/
│   ├── package.json
│   └── ... (all files)
└── diffs/                 # Diff storage for webhooks

5. File Storage in Database

Files are stored using the optimized JSON schema (migration 003):

Table: repository_files

• One row per directory
• All files in a directory stored as JSON array
• Unique constraint on directory_id

Example:

{
  "directory_id": "uuid-123",
  "relative_path": "src/components",
  "files": [
    {
      "filename": "Header.tsx",
      "file_extension": ".tsx",
      "file_size_bytes": 1024,
      "is_binary": false,
      "mime_type": "text/plain"
    },
    {
      "filename": "Footer.tsx",
      "file_extension": ".tsx",
      "file_size_bytes": 512,
      "is_binary": false,
      "mime_type": "text/plain"
    }
  ]
}

6. Automatic Webhook Creation

After successful repository attachment, the service automatically creates a webhook:

Webhook Configuration:

• URL: https://98fa7f21e2e9.ngrok-free.app/api/github/webhook
• Secret: mywebhooksecret2025
• Events: push
• Content Type: application/json

Webhook Endpoint: POST /api/github/webhook

What happens on push:

1. GitHub sends webhook payload
2. Service verifies signature using webhook secret
3. Processes commit changes
4. Updates repository files in database
5. Stores diff information

7. Response to Frontend

Success Response (201):

{
  "success": true,
  "message": "Repository attached and synced successfully",
  "data": {
    "repository_id": "uuid-456",
    "repository_name": "repo",
    "owner_name": "owner",
    "branch_name": "main",
    "is_public": true,
    "sync_status": "synced",
    "webhook_result": {
      "created": true,
      "hook_id": 12345
    },
    "storage_info": {
      "total_files": 150,
      "total_directories": 25,
      "total_size_bytes": 1048576
    }
  }
}

Auth Required Response (401):

{
  "success": false,
  "message": "GitHub authentication required for private repository",
  "requires_auth": true,
  "auth_url": "http://localhost:8000/api/github/auth/github?redirect=1&state=...",
  "repository_info": {
    "owner": "owner",
    "repo": "repo",
    "repository_url": "https://github.com/owner/repo",
    "branch_name": "main"
  }
}

Frontend Integration Guide

Step 1: Attach Repository Request

const attachRepository = async (repoUrl: string, branch: string, userId: string) => {
  const response = await fetch('http://localhost:8000/api/github/attach-repository', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'x-user-id': userId
    },
    body: JSON.stringify({
      repository_url: repoUrl,
      branch_name: branch
    })
  });

  const data = await response.json();
  
  if (response.status === 401 && data.requires_auth) {
    // Redirect to OAuth
    window.location.href = data.auth_url;
  } else if (response.status === 201) {
    // Success - repository attached
    console.log('Repository attached:', data.data);
  }
};

Step 2: Handle OAuth Redirect

// In your /project-builder page
useEffect(() => {
  const params = new URLSearchParams(window.location.search);
  
  if (params.get('github_connected') === '1') {
    // GitHub connected successfully
    const repoAttached = params.get('repo_attached') === '1';
    const repositoryId = params.get('repository_id');
    const syncStatus = params.get('sync_status');
    
    if (repoAttached) {
      // Repository was auto-attached after OAuth
      console.log('Repository attached:', repositoryId, syncStatus);
    }
  }
}, []);

Step 3: Display Repository Data

const getRepositoryFiles = async (repositoryId: string) => {
  const response = await fetch(
    `http://localhost:8000/api/github/repository/${repositoryId}/files`
  );
  
  const data = await response.json();
  return data.data; // File structure
};

Database Schema

Main Tables

1. all_repositories: Repository metadata
2. repository_storage: Storage tracking
3. repository_directories: Directory structure
4. repository_files: Files as JSON arrays (one row per directory)
5. github_user_tokens: OAuth tokens
6. github_webhooks: Webhook event logs

Key Features

✅ Public/Private Detection: Automatic detection and handling ✅ OAuth Flow: Seamless GitHub authentication ✅ Full Git Clone: Includes .git directory and complete history ✅ Automatic Webhooks: Auto-creates webhooks with ngrok URL ✅ JSON Storage: Optimized file storage in database ✅ Auto-Attach: Repository automatically attached after OAuth ✅ Frontend Redirect: Redirects back to your app after OAuth

Testing the Flow

1. Start the services:

   docker compose up -d --build git-integration
   

2. Try attaching a public repository:

   curl -X POST http://localhost:8000/api/github/attach-repository \
     -H "Content-Type: application/json" \
     -H "x-user-id: your-user-id" \
     -d '{"repository_url": "https://github.com/octocat/Hello-World", "branch_name": "master"}'
   

3. Try attaching a private repository:

   • Will return auth_url
   • Visit the auth_url in browser
   • Authorize on GitHub
   • Will redirect back and auto-attach

4. Check stored files:

   ls -la "/home/tech4biz/Desktop/today work/git-repo/"
   

5. Verify webhook was created:

   • Go to GitHub repository → Settings → Webhooks
   • Should see webhook with your ngrok URL

Troubleshooting

Repository not cloning

• Check ATTACHED_REPOS_DIR permissions
• Verify OAuth token is valid
• Check git-integration service logs

Webhook not created

• Verify PUBLIC_BASE_URL is set correctly
• Check OAuth token has admin:repo_hook scope
• Verify ngrok is running and accessible

OAuth redirect not working

• Check FRONTEND_URL environment variable
• Verify GITHUB_REDIRECT_URI matches GitHub App settings
• Check browser console for errors