536 lines
16 KiB
Markdown
536 lines
16 KiB
Markdown
# Data Collection Analysis - What We Have vs What We're Collecting
|
|
|
|
## Overview
|
|
This document compares the database structure with what we're currently collecting and recommends what we should start collecting for the Detailed Reports.
|
|
|
|
---
|
|
|
|
## 1. ACTIVITIES TABLE
|
|
|
|
### ✅ **Database Fields Available:**
|
|
```sql
|
|
- activity_id (PK)
|
|
- request_id (FK) ✅ COLLECTING
|
|
- user_id (FK) ✅ COLLECTING
|
|
- user_name ✅ COLLECTING
|
|
- activity_type ✅ COLLECTING
|
|
- activity_description ✅ COLLECTING
|
|
- activity_category ❌ NOT COLLECTING (set to NULL)
|
|
- severity ❌ NOT COLLECTING (set to NULL)
|
|
- metadata ✅ COLLECTING (partially)
|
|
- is_system_event ✅ COLLECTING
|
|
- ip_address ❌ NOT COLLECTING (set to NULL)
|
|
- user_agent ❌ NOT COLLECTING (set to NULL)
|
|
- created_at ✅ COLLECTING
|
|
```
|
|
|
|
### 🔴 **Currently NOT Collecting (But Should):**
|
|
|
|
1. **IP Address** (`ip_address`)
|
|
- **Status:** Field exists, but always set to `null`
|
|
- **Impact:** Cannot show IP in User Activity Log Report
|
|
- **Fix:** Extract from `req.ip` or `req.headers['x-forwarded-for']` in controllers
|
|
- **Priority:** HIGH (needed for security/audit)
|
|
|
|
2. **User Agent** (`user_agent`)
|
|
- **Status:** Field exists, but always set to `null`
|
|
- **Impact:** Cannot show device/browser info in reports
|
|
- **Fix:** Extract from `req.headers['user-agent']` in controllers
|
|
- **Priority:** MEDIUM (nice to have for analytics)
|
|
|
|
3. **Activity Category** (`activity_category`)
|
|
- **Status:** Field exists, but always set to `null`
|
|
- **Impact:** Cannot categorize activities (e.g., "AUTHENTICATION", "WORKFLOW", "DOCUMENT")
|
|
- **Fix:** Map `activity_type` to category:
|
|
- `created`, `approval`, `rejection`, `status_change` → "WORKFLOW"
|
|
- `comment` → "COLLABORATION"
|
|
- `document_added` → "DOCUMENT"
|
|
- `sla_warning` → "SYSTEM"
|
|
- **Priority:** MEDIUM (helps with filtering/reporting)
|
|
|
|
4. **Severity** (`severity`)
|
|
- **Status:** Field exists, but always set to `null`
|
|
- **Impact:** Cannot prioritize critical activities
|
|
- **Fix:** Map based on activity type:
|
|
- `rejection`, `sla_warning` → "WARNING"
|
|
- `approval`, `closed` → "INFO"
|
|
- `status_change` → "INFO"
|
|
- **Priority:** LOW (optional enhancement)
|
|
|
|
### 📝 **Recommendation:**
|
|
**Update `activity.service.ts` to accept and store:**
|
|
```typescript
|
|
async log(entry: ActivityEntry & {
|
|
ipAddress?: string;
|
|
userAgent?: string;
|
|
category?: string;
|
|
severity?: string;
|
|
}) {
|
|
// ... existing code ...
|
|
const activityData = {
|
|
// ... existing fields ...
|
|
ipAddress: entry.ipAddress || null,
|
|
userAgent: entry.userAgent || null,
|
|
activityCategory: entry.category || this.inferCategory(entry.type),
|
|
severity: entry.severity || this.inferSeverity(entry.type),
|
|
};
|
|
}
|
|
```
|
|
|
|
**Update all controller calls to pass IP and User Agent:**
|
|
```typescript
|
|
activityService.log({
|
|
// ... existing fields ...
|
|
ipAddress: req.ip || req.headers['x-forwarded-for'] || null,
|
|
userAgent: req.headers['user-agent'] || null,
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## 2. APPROVAL_LEVELS TABLE
|
|
|
|
### ✅ **Database Fields Available:**
|
|
```sql
|
|
- level_id (PK)
|
|
- request_id (FK) ✅ COLLECTING
|
|
- level_number ✅ COLLECTING
|
|
- level_name ❌ OPTIONAL (may not be set)
|
|
- approver_id (FK) ✅ COLLECTING
|
|
- approver_email ✅ COLLECTING
|
|
- approver_name ✅ COLLECTING
|
|
- tat_hours ✅ COLLECTING
|
|
- tat_days ✅ COLLECTING (auto-calculated)
|
|
- status ✅ COLLECTING
|
|
- level_start_time ✅ COLLECTING
|
|
- level_end_time ✅ COLLECTING
|
|
- action_date ✅ COLLECTING
|
|
- comments ✅ COLLECTING
|
|
- rejection_reason ✅ COLLECTING
|
|
- is_final_approver ✅ COLLECTING
|
|
- elapsed_hours ✅ COLLECTING
|
|
- remaining_hours ✅ COLLECTING
|
|
- tat_percentage_used ✅ COLLECTING
|
|
- tat50_alert_sent ✅ COLLECTING
|
|
- tat75_alert_sent ✅ COLLECTING
|
|
- tat_breached ✅ COLLECTING
|
|
- tat_start_time ✅ COLLECTING
|
|
- created_at ✅ COLLECTING
|
|
- updated_at ✅ COLLECTING
|
|
```
|
|
|
|
### 🔴 **Currently NOT Collecting (But Should):**
|
|
|
|
1. **Level Name** (`level_name`)
|
|
- **Status:** Field exists, but may be NULL
|
|
- **Impact:** Cannot show stage name in reports (only level number)
|
|
- **Fix:** When creating approval levels, prompt for or auto-generate level names:
|
|
- "Department Head Review"
|
|
- "Finance Approval"
|
|
- "Final Approval"
|
|
- **Priority:** MEDIUM (improves report readability)
|
|
|
|
### 📝 **Recommendation:**
|
|
**Ensure level_name is set when creating approval levels:**
|
|
```typescript
|
|
await ApprovalLevel.create({
|
|
// ... existing fields ...
|
|
levelName: levelData.levelName || `Level ${levelNumber}`,
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## 3. USER_SESSIONS TABLE
|
|
|
|
### ✅ **Database Fields Available:**
|
|
```sql
|
|
- session_id (PK)
|
|
- user_id (FK)
|
|
- session_token ✅ COLLECTING
|
|
- refresh_token ✅ COLLECTING
|
|
- ip_address ❓ CHECK IF COLLECTING
|
|
- user_agent ❓ CHECK IF COLLECTING
|
|
- device_type ❓ CHECK IF COLLECTING
|
|
- browser ❓ CHECK IF COLLECTING
|
|
- os ❓ CHECK IF COLLECTING
|
|
- login_at ✅ COLLECTING
|
|
- last_activity_at ✅ COLLECTING
|
|
- logout_at ❓ CHECK IF COLLECTING
|
|
- expires_at ✅ COLLECTING
|
|
- is_active ✅ COLLECTING
|
|
- logout_reason ❓ CHECK IF COLLECTING
|
|
```
|
|
|
|
### 🔴 **Missing for Login Activity Tracking:**
|
|
|
|
1. **Login Activities in Activities Table**
|
|
- **Status:** Login events are NOT logged in `activities` table
|
|
- **Impact:** Cannot show login activities in User Activity Log Report
|
|
- **Fix:** Add login activity logging in auth middleware/controller:
|
|
```typescript
|
|
// After successful login
|
|
await activityService.log({
|
|
requestId: 'SYSTEM_LOGIN', // Special request ID for system events
|
|
type: 'login',
|
|
user: { userId, name: user.displayName },
|
|
ipAddress: req.ip,
|
|
userAgent: req.headers['user-agent'],
|
|
category: 'AUTHENTICATION',
|
|
severity: 'INFO',
|
|
timestamp: new Date().toISOString(),
|
|
action: 'User Login',
|
|
details: `User logged in from ${req.ip}`
|
|
});
|
|
```
|
|
- **Priority:** HIGH (needed for security audit)
|
|
|
|
2. **Device/Browser Parsing**
|
|
- **Status:** Fields exist but may not be populated
|
|
- **Impact:** Cannot show device type in reports
|
|
- **Fix:** Parse user agent to extract:
|
|
- `device_type`: "WEB", "MOBILE"
|
|
- `browser`: "Chrome", "Firefox", "Safari"
|
|
- `os`: "Windows", "macOS", "iOS", "Android"
|
|
- **Priority:** MEDIUM (nice to have)
|
|
|
|
---
|
|
|
|
## 4. WORKFLOW_REQUESTS TABLE
|
|
|
|
### ✅ **All Fields Are Being Collected:**
|
|
- All fields in `workflow_requests` are properly collected
|
|
- No missing data here
|
|
|
|
### 📝 **Note:**
|
|
- `submission_date` vs `created_at`: Use `submission_date` for "days open" calculation
|
|
- `closure_date`: Available for completed requests
|
|
|
|
---
|
|
|
|
## 5. TAT_TRACKING TABLE
|
|
|
|
### ✅ **Database Fields Available:**
|
|
```sql
|
|
- tracking_id (PK)
|
|
- request_id (FK)
|
|
- level_id (FK)
|
|
- tracking_type ✅ COLLECTING
|
|
- tat_status ✅ COLLECTING
|
|
- total_tat_hours ✅ COLLECTING
|
|
- elapsed_hours ✅ COLLECTING
|
|
- remaining_hours ✅ COLLECTING
|
|
- percentage_used ✅ COLLECTING
|
|
- threshold_50_breached ✅ COLLECTING
|
|
- threshold_50_alerted_at ✅ COLLECTING
|
|
- threshold_80_breached ✅ COLLECTING
|
|
- threshold_80_alerted_at ✅ COLLECTING
|
|
- threshold_100_breached ✅ COLLECTING
|
|
- threshold_100_alerted_at ✅ COLLECTING
|
|
- alert_count ✅ COLLECTING
|
|
- last_calculated_at ✅ COLLECTING
|
|
```
|
|
|
|
### ✅ **All Fields Are Being Collected:**
|
|
- TAT tracking appears to be fully implemented
|
|
|
|
---
|
|
|
|
## 6. AUDIT_LOGS TABLE
|
|
|
|
### ✅ **Database Fields Available:**
|
|
```sql
|
|
- audit_id (PK)
|
|
- user_id (FK)
|
|
- entity_type
|
|
- entity_id
|
|
- action
|
|
- action_category
|
|
- old_values (JSONB)
|
|
- new_values (JSONB)
|
|
- changes_summary
|
|
- ip_address
|
|
- user_agent
|
|
- session_id
|
|
- request_method
|
|
- request_url
|
|
- response_status
|
|
- execution_time_ms
|
|
- created_at
|
|
```
|
|
|
|
### 🔴 **Status:**
|
|
- **Audit logging may not be fully implemented**
|
|
- **Impact:** Cannot track all system changes for audit purposes
|
|
- **Priority:** MEDIUM (for compliance/security)
|
|
|
|
---
|
|
|
|
## SUMMARY: What to Start Collecting
|
|
|
|
### 🔴 **HIGH PRIORITY (Must Have for Reports):**
|
|
|
|
1. **IP Address in Activities** ✅ Field exists, just need to populate
|
|
- Extract from `req.ip` or `req.headers['x-forwarded-for']`
|
|
- Update `activity.service.ts` to accept IP
|
|
- Update all controller calls
|
|
|
|
2. **User Agent in Activities** ✅ Field exists, just need to populate
|
|
- Extract from `req.headers['user-agent']`
|
|
- Update `activity.service.ts` to accept user agent
|
|
- Update all controller calls
|
|
|
|
3. **Login Activities** ❌ Not currently logged
|
|
- Add login activity logging in auth controller
|
|
- Use special `requestId: 'SYSTEM_LOGIN'` for system events
|
|
- Include IP and user agent
|
|
|
|
### 🟡 **MEDIUM PRIORITY (Nice to Have):**
|
|
|
|
4. **Activity Category** ✅ Field exists, just need to populate
|
|
- Auto-infer from `activity_type`
|
|
- Helps with filtering and reporting
|
|
|
|
5. **Level Names** ✅ Field exists, ensure it's set
|
|
- Improve readability in reports
|
|
- Auto-generate if not provided
|
|
|
|
6. **Severity** ✅ Field exists, just need to populate
|
|
- Auto-infer from `activity_type`
|
|
- Helps prioritize critical activities
|
|
|
|
### 🟢 **LOW PRIORITY (Future Enhancement):**
|
|
|
|
7. **Device/Browser Parsing**
|
|
- Parse user agent to extract device type, browser, OS
|
|
- Store in `user_sessions` table
|
|
|
|
8. **Audit Logging**
|
|
- Implement comprehensive audit logging
|
|
- Track all system changes
|
|
|
|
---
|
|
|
|
## 7. BUSINESS DAYS CALCULATION FOR WORKFLOW AGING
|
|
|
|
### ✅ **Available:**
|
|
- `calculateElapsedWorkingHours()` - Calculates working hours (excludes weekends/holidays)
|
|
- Working hours configuration (9 AM - 6 PM, Mon-Fri)
|
|
- Holiday support (from database)
|
|
- Priority-based calculation (express vs standard)
|
|
|
|
### ❌ **Missing:**
|
|
1. **Business Days Count Function**
|
|
- Need a function to calculate business days (not hours)
|
|
- For Workflow Aging Report: "Days Open" should be business days
|
|
- Currently only have working hours calculation
|
|
|
|
2. **TAT Processor Using Wrong Calculation**
|
|
- `tatProcessor.ts` uses simple calendar hours:
|
|
```typescript
|
|
const elapsedMs = now.getTime() - new Date(levelStartTime).getTime();
|
|
const elapsedHours = elapsedMs / (1000 * 60 * 60);
|
|
```
|
|
- Should use `calculateElapsedWorkingHours()` instead
|
|
- This causes incorrect TAT breach calculations
|
|
|
|
### 🔧 **What Needs to be Built:**
|
|
|
|
1. **Add Business Days Calculation Function:**
|
|
```typescript
|
|
// In tatTimeUtils.ts
|
|
export async function calculateBusinessDays(
|
|
startDate: Date | string,
|
|
endDate: Date | string = new Date(),
|
|
priority: string = 'standard'
|
|
): Promise<number> {
|
|
await loadWorkingHoursCache();
|
|
await loadHolidaysCache();
|
|
|
|
let start = dayjs(startDate);
|
|
const end = dayjs(endDate);
|
|
const config = workingHoursCache || { /* defaults */ };
|
|
|
|
let businessDays = 0;
|
|
let current = start.startOf('day');
|
|
|
|
while (current.isBefore(end) || current.isSame(end, 'day')) {
|
|
const dayOfWeek = current.day();
|
|
const dateStr = current.format('YYYY-MM-DD');
|
|
|
|
const isWorkingDay = priority === 'express'
|
|
? true
|
|
: (dayOfWeek >= config.startDay && dayOfWeek <= config.endDay);
|
|
const isNotHoliday = !holidaysCache.has(dateStr);
|
|
|
|
if (isWorkingDay && isNotHoliday) {
|
|
businessDays++;
|
|
}
|
|
|
|
current = current.add(1, 'day');
|
|
}
|
|
|
|
return businessDays;
|
|
}
|
|
```
|
|
|
|
2. **Fix TAT Processor:**
|
|
- Replace calendar hours calculation with `calculateElapsedWorkingHours()`
|
|
- This will fix TAT breach alerts to use proper working hours
|
|
|
|
3. **Update Workflow Aging Report:**
|
|
- Use `calculateBusinessDays()` instead of calendar days
|
|
- Filter by business days threshold
|
|
|
|
---
|
|
|
|
## IMPLEMENTATION CHECKLIST
|
|
|
|
### Phase 1: Quick Wins (Fields Exist, Just Need to Populate)
|
|
- [ ] Update `activity.service.ts` to accept `ipAddress` and `userAgent`
|
|
- [ ] Update all controller calls to pass IP and user agent
|
|
- [ ] Add activity category inference
|
|
- [ ] Add severity inference
|
|
|
|
### Phase 2: Fix TAT Calculations (CRITICAL)
|
|
- [x] Fix `tatProcessor.ts` to use `calculateElapsedWorkingHours()` instead of calendar hours ✅
|
|
- [x] Add `calculateBusinessDays()` function to `tatTimeUtils.ts` ✅
|
|
- [ ] Test TAT breach calculations with working hours
|
|
|
|
### Phase 3: New Functionality
|
|
- [x] Add login activity logging ✅ (Implemented in auth.controller.ts for SSO and token exchange)
|
|
- [x] Ensure level names are set when creating approval levels ✅ (levelName set in workflow.service.ts)
|
|
- [x] Add device/browser parsing for user sessions ✅ (userAgentParser.ts utility created - can be used for parsing user agent strings)
|
|
|
|
### Phase 4: Enhanced Reporting
|
|
- [x] Build report endpoints using collected data ✅ (getLifecycleReport, getActivityLogReport, getWorkflowAgingReport)
|
|
- [x] Add filtering by category, severity ✅ (Filtering by category and severity added to getActivityLogReport, frontend UI added)
|
|
- [x] Add IP/user agent to activity log reports ✅ (IP and user agent captured and displayed)
|
|
- [x] Use business days in Workflow Aging Report ✅ (calculateBusinessDays implemented and used)
|
|
|
|
---
|
|
|
|
## CODE CHANGES NEEDED
|
|
|
|
### 1. Update Activity Service (`activity.service.ts`)
|
|
|
|
```typescript
|
|
export type ActivityEntry = {
|
|
requestId: string;
|
|
type: 'created' | 'assignment' | 'approval' | 'rejection' | 'status_change' | 'comment' | 'reminder' | 'document_added' | 'sla_warning' | 'ai_conclusion_generated' | 'closed' | 'login';
|
|
user?: { userId: string; name?: string; email?: string };
|
|
timestamp: string;
|
|
action: string;
|
|
details: string;
|
|
metadata?: any;
|
|
ipAddress?: string; // NEW
|
|
userAgent?: string; // NEW
|
|
category?: string; // NEW
|
|
severity?: string; // NEW
|
|
};
|
|
|
|
class ActivityService {
|
|
private inferCategory(type: string): string {
|
|
const categoryMap: Record<string, string> = {
|
|
'created': 'WORKFLOW',
|
|
'approval': 'WORKFLOW',
|
|
'rejection': 'WORKFLOW',
|
|
'status_change': 'WORKFLOW',
|
|
'assignment': 'WORKFLOW',
|
|
'comment': 'COLLABORATION',
|
|
'document_added': 'DOCUMENT',
|
|
'sla_warning': 'SYSTEM',
|
|
'reminder': 'SYSTEM',
|
|
'ai_conclusion_generated': 'SYSTEM',
|
|
'closed': 'WORKFLOW',
|
|
'login': 'AUTHENTICATION'
|
|
};
|
|
return categoryMap[type] || 'OTHER';
|
|
}
|
|
|
|
private inferSeverity(type: string): string {
|
|
const severityMap: Record<string, string> = {
|
|
'rejection': 'WARNING',
|
|
'sla_warning': 'WARNING',
|
|
'approval': 'INFO',
|
|
'closed': 'INFO',
|
|
'status_change': 'INFO',
|
|
'login': 'INFO',
|
|
'created': 'INFO',
|
|
'comment': 'INFO',
|
|
'document_added': 'INFO'
|
|
};
|
|
return severityMap[type] || 'INFO';
|
|
}
|
|
|
|
async log(entry: ActivityEntry) {
|
|
// ... existing code ...
|
|
const activityData = {
|
|
requestId: entry.requestId,
|
|
userId: entry.user?.userId || null,
|
|
userName: entry.user?.name || entry.user?.email || null,
|
|
activityType: entry.type,
|
|
activityDescription: entry.details,
|
|
activityCategory: entry.category || this.inferCategory(entry.type),
|
|
severity: entry.severity || this.inferSeverity(entry.type),
|
|
metadata: entry.metadata || null,
|
|
isSystemEvent: !entry.user,
|
|
ipAddress: entry.ipAddress || null, // NEW
|
|
userAgent: entry.userAgent || null, // NEW
|
|
};
|
|
// ... rest of code ...
|
|
}
|
|
}
|
|
```
|
|
|
|
### 2. Update Controller Calls (Example)
|
|
|
|
```typescript
|
|
// In workflow.controller.ts, approval.controller.ts, etc.
|
|
activityService.log({
|
|
requestId: workflow.requestId,
|
|
type: 'created',
|
|
user: { userId, name: user.displayName },
|
|
timestamp: new Date().toISOString(),
|
|
action: 'Request Created',
|
|
details: `Request ${workflow.requestNumber} created`,
|
|
ipAddress: req.ip || req.headers['x-forwarded-for'] || null, // NEW
|
|
userAgent: req.headers['user-agent'] || null, // NEW
|
|
});
|
|
```
|
|
|
|
### 3. Add Login Activity Logging
|
|
|
|
```typescript
|
|
// In auth.controller.ts after successful login
|
|
await activityService.log({
|
|
requestId: 'SYSTEM_LOGIN', // Special ID for system events
|
|
type: 'login',
|
|
user: { userId: user.userId, name: user.displayName },
|
|
timestamp: new Date().toISOString(),
|
|
action: 'User Login',
|
|
details: `User logged in successfully`,
|
|
ipAddress: req.ip || req.headers['x-forwarded-for'] || null,
|
|
userAgent: req.headers['user-agent'] || null,
|
|
category: 'AUTHENTICATION',
|
|
severity: 'INFO'
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## CONCLUSION
|
|
|
|
**Good News:** Most fields already exist in the database! We just need to:
|
|
1. Populate existing fields (IP, user agent, category, severity)
|
|
2. Add login activity logging
|
|
3. Ensure level names are set
|
|
|
|
**Estimated Effort:**
|
|
- Phase 1 (Quick Wins): 2-4 hours
|
|
- Phase 2 (New Functionality): 4-6 hours
|
|
- Phase 3 (Enhanced Reporting): 8-12 hours
|
|
|
|
**Total: ~14-22 hours of development work**
|
|
|