codenuk_backend_mine/services/git-integration/FLOW_DOCUMENTATION.md

# Git Integration Service - Complete Flow Documentation

## Overview
This document describes the complete flow of the git-integration service for attaching GitHub repositories.

## Configuration

### Environment Variables
- `GITHUB_CLIENT_ID`: OAuth App Client ID
- `GITHUB_CLIENT_SECRET`: OAuth App Client Secret
- `GITHUB_REDIRECT_URI`: OAuth callback URL (http://localhost:8000/api/github/auth/github/callback)
- `GITHUB_WEBHOOK_SECRET`: mywebhooksecret2025
- `PUBLIC_BASE_URL`: https://98fa7f21e2e9.ngrok-free.app (your ngrok URL)
- `ATTACHED_REPOS_DIR`: /app/git-repos
- `DIFF_STORAGE_DIR`: /app/git-repos/diffs
- `FRONTEND_URL`: http://localhost:3001

### Storage Location
- **Container Path**: `/app/git-repos`
- **Host Path**: `/home/tech4biz/Desktop/today work/git-repo`
- **Repository Format**: `owner__REPONAME__branch/`

## Complete User Flow

### 1. User Attaches Repository

**Endpoint**: `POST /api/github/attach-repository`

**Request Body**:
```json
{
  "repository_url": "https://github.com/owner/repo",
  "branch_name": "main"
}
```

**Headers**:
- `x-user-id`: User's unique ID

### 2. Public/Private Detection

The service automatically detects if the repository is public or private:

#### For Public Repositories:
1. ✅ Attempts unauthenticated access via GitHub API
2. ✅ If successful, proceeds to clone without OAuth
3. ✅ Clones repository using `git clone` (full clone with .git directory)
4. ✅ Stores all files including .git folder
5. ✅ Creates webhook automatically (if PUBLIC_BASE_URL is set)

#### For Private Repositories:
1. ❌ Unauthenticated access fails (404 error)
2. 🔐 Checks if user has GitHub OAuth token
3. **If NOT authenticated**:
   - Returns 401 with OAuth URL
   - User is redirected to GitHub for authentication
4. **If authenticated**:
   - Uses OAuth token to access repository
   - Proceeds to clone with authentication

### 3. GitHub OAuth Flow (Private Repos Only)

**Step 1: Initiate OAuth**
- Frontend receives auth_url from backend
- User clicks "Connect GitHub" button
- Redirects to: `http://localhost:8000/api/github/auth/github?redirect=1&user_id={userId}&state={state}`

**Step 2: GitHub Authorization**
- User authorizes the app on GitHub
- GitHub redirects to callback URL with authorization code

**Step 3: OAuth Callback**
- Endpoint: `GET /api/github/auth/github/callback`
- Exchanges code for access token
- Stores token in database (table: `github_user_tokens`)
- **Auto-attaches repository** if state contains repo context
- Redirects back to frontend: `{FRONTEND_URL}/project-builder?github_connected=1&repo_attached=1`

### 4. Repository Cloning

**Clone Method**: Full git clone (includes .git directory)

**For Public Repos**:
```bash
git clone -b {branch} https://github.com/{owner}/{repo}.git
```

**For Private Repos**:
```bash
git clone -b {branch} https://oauth2:{token}@github.com/{owner}/{repo}.git
```

**Storage Structure**:
```
/home/tech4biz/Desktop/today work/git-repo/
├── owner__REPONAME__branch/
│   ├── .git/              # ✅ Full git history included
│   ├── src/
│   ├── package.json
│   └── ... (all files)
└── diffs/                 # Diff storage for webhooks
```

### 5. File Storage in Database

Files are stored using the **optimized JSON schema** (migration 003):

**Table: `repository_files`**
- One row per directory
- All files in a directory stored as JSON array
- Unique constraint on `directory_id`

**Example**:
```json
{
  "directory_id": "uuid-123",
  "relative_path": "src/components",
  "files": [
    {
      "filename": "Header.tsx",
      "file_extension": ".tsx",
      "file_size_bytes": 1024,
      "is_binary": false,
      "mime_type": "text/plain"
    },
    {
      "filename": "Footer.tsx",
      "file_extension": ".tsx",
      "file_size_bytes": 512,
      "is_binary": false,
      "mime_type": "text/plain"
    }
  ]
}
```

### 6. Automatic Webhook Creation

After successful repository attachment, the service automatically creates a webhook:

**Webhook Configuration**:
- **URL**: `https://98fa7f21e2e9.ngrok-free.app/api/github/webhook`
- **Secret**: `mywebhooksecret2025`
- **Events**: `push`
- **Content Type**: `application/json`

**Webhook Endpoint**: `POST /api/github/webhook`

**What happens on push**:
1. GitHub sends webhook payload
2. Service verifies signature using webhook secret
3. Processes commit changes
4. Updates repository files in database
5. Stores diff information

### 7. Response to Frontend

**Success Response** (201):
```json
{
  "success": true,
  "message": "Repository attached and synced successfully",
  "data": {
    "repository_id": "uuid-456",
    "repository_name": "repo",
    "owner_name": "owner",
    "branch_name": "main",
    "is_public": true,
    "sync_status": "synced",
    "webhook_result": {
      "created": true,
      "hook_id": 12345
    },
    "storage_info": {
      "total_files": 150,
      "total_directories": 25,
      "total_size_bytes": 1048576
    }
  }
}
```

**Auth Required Response** (401):
```json
{
  "success": false,
  "message": "GitHub authentication required for private repository",
  "requires_auth": true,
  "auth_url": "http://localhost:8000/api/github/auth/github?redirect=1&state=...",
  "repository_info": {
    "owner": "owner",
    "repo": "repo",
    "repository_url": "https://github.com/owner/repo",
    "branch_name": "main"
  }
}
```

## Frontend Integration Guide

### Step 1: Attach Repository Request

```typescript
const attachRepository = async (repoUrl: string, branch: string, userId: string) => {
  const response = await fetch('http://localhost:8000/api/github/attach-repository', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'x-user-id': userId
    },
    body: JSON.stringify({
      repository_url: repoUrl,
      branch_name: branch
    })
  });

  const data = await response.json();
  
  if (response.status === 401 && data.requires_auth) {
    // Redirect to OAuth
    window.location.href = data.auth_url;
  } else if (response.status === 201) {
    // Success - repository attached
    console.log('Repository attached:', data.data);
  }
};
```

### Step 2: Handle OAuth Redirect

```typescript
// In your /project-builder page
useEffect(() => {
  const params = new URLSearchParams(window.location.search);
  
  if (params.get('github_connected') === '1') {
    // GitHub connected successfully
    const repoAttached = params.get('repo_attached') === '1';
    const repositoryId = params.get('repository_id');
    const syncStatus = params.get('sync_status');
    
    if (repoAttached) {
      // Repository was auto-attached after OAuth
      console.log('Repository attached:', repositoryId, syncStatus);
    }
  }
}, []);
```

### Step 3: Display Repository Data

```typescript
const getRepositoryFiles = async (repositoryId: string) => {
  const response = await fetch(
    `http://localhost:8000/api/github/repository/${repositoryId}/files`
  );
  
  const data = await response.json();
  return data.data; // File structure
};
```

## Database Schema

### Main Tables

1. **all_repositories**: Repository metadata
2. **repository_storage**: Storage tracking
3. **repository_directories**: Directory structure
4. **repository_files**: Files as JSON arrays (one row per directory)
5. **github_user_tokens**: OAuth tokens
6. **github_webhooks**: Webhook event logs

## Key Features

✅ **Public/Private Detection**: Automatic detection and handling
✅ **OAuth Flow**: Seamless GitHub authentication
✅ **Full Git Clone**: Includes .git directory and complete history
✅ **Automatic Webhooks**: Auto-creates webhooks with ngrok URL
✅ **JSON Storage**: Optimized file storage in database
✅ **Auto-Attach**: Repository automatically attached after OAuth
✅ **Frontend Redirect**: Redirects back to your app after OAuth

## Testing the Flow

1. Start the services:
   ```bash
   docker compose up -d --build git-integration
   ```

2. Try attaching a public repository:
   ```bash
   curl -X POST http://localhost:8000/api/github/attach-repository \
     -H "Content-Type: application/json" \
     -H "x-user-id: your-user-id" \
     -d '{"repository_url": "https://github.com/octocat/Hello-World", "branch_name": "master"}'
   ```

3. Try attaching a private repository:
   - Will return auth_url
   - Visit the auth_url in browser
   - Authorize on GitHub
   - Will redirect back and auto-attach

4. Check stored files:
   ```bash
   ls -la "/home/tech4biz/Desktop/today work/git-repo/"
   ```

5. Verify webhook was created:
   - Go to GitHub repository → Settings → Webhooks
   - Should see webhook with your ngrok URL

## Troubleshooting

### Repository not cloning
- Check ATTACHED_REPOS_DIR permissions
- Verify OAuth token is valid
- Check git-integration service logs

### Webhook not created
- Verify PUBLIC_BASE_URL is set correctly
- Check OAuth token has admin:repo_hook scope
- Verify ngrok is running and accessible

### OAuth redirect not working
- Check FRONTEND_URL environment variable
- Verify GITHUB_REDIRECT_URI matches GitHub App settings
- Check browser console for errors