Troubleshooting

This guide covers common issues, diagnostic steps, and solutions for Terragnos Core deployments.

Common Issues

API Won't Start

Symptoms:

• API container exits immediately
• Health checks fail
• No response from API endpoints

Diagnostic steps:

1. Check container logs:

   docker compose logs api

2. Verify environment variables:

   docker compose config

3. Check database connectivity:

   docker compose exec api env | grep DATABASE_URL

Common causes:

• Missing or invalid DATABASE_URL
• Database not accessible
• Invalid AUTH_JWT_SECRET format
• Port already in use

Database Connection Errors

Symptoms:

• ECONNREFUSED errors in logs
• API returns 500 errors
• Health check shows database as unhealthy

Diagnostic steps:

1. Verify database is running:

   docker compose ps db

2. Test database connection:

   docker compose exec db psql -U terragnos -d terragnos -c "SELECT 1;"

3. Check database logs:

   docker compose logs db

Common causes:

• Database container not started
• Incorrect DATABASE_URL
• Database password mismatch
• Network connectivity issues

License Errors

Symptoms:

• 403 License limit exceeded responses
• License-related errors in logs
• Features not available

Diagnostic steps:

1. Check license status:

   curl -H "Authorization: Bearer <token>" \
     http://localhost:3000/v1/license

2. Verify license environment variables:

   docker compose exec api env | grep LICENSE

3. Check license expiration:

   {
     "status": "expired",
     "expiresAt": "2024-01-01T00:00:00Z"
   }

Common causes:

• License expired
• License limit reached
• Invalid license key
• Missing LICENSE_PUBLIC_KEY

Authentication Failures

Symptoms:

• 401 Unauthorized responses
• Cannot access API endpoints
• Token validation errors

Diagnostic steps:

1. Verify JWT token format:

   # Decode JWT (without verification)
   echo "<token>" | cut -d. -f2 | base64 -d

2. Check token expiration:

   {
     "exp": 1704153600  // Unix timestamp
   }

3. Verify AUTH_JWT_SECRET matches token issuer

Common causes:

• Expired token
• Invalid token signature
• Missing or incorrect AUTH_JWT_SECRET
• Token missing required claims (tenant, sub)

Workflow Transition Failures

Symptoms:

• 403 Forbidden on workflow transitions
• Transitions blocked by guards
• Workflow state not updating

Diagnostic steps:

1. Check available transitions:

   curl -H "Authorization: Bearer <token>" \
     http://localhost:3000/v1/objects/{id}/workflow/explain-transitions

2. Review guard requirements:

   {
     "availableTransitions": [
       {
         "targetState": "approved",
         "guards": [
           {
             "type": "permission",
             "requiredRole": "admin",
             "evaluated": false,
             "reason": "User does not have admin role"
           }
         ]
       }
     ]
   }

Common causes:

• Insufficient permissions
• Guard conditions not met
• Object in invalid state
• Workflow version mismatch

Automation Rule Not Executing

Symptoms:

• Automation rules not triggering
• No execution logs
• Expected effects not happening

Diagnostic steps:

1. Check rule status:

   curl -H "Authorization: Bearer <token>" \
     http://localhost:3000/v1/automation/rules/{ruleId}

2. View execution history:

   curl -H "Authorization: Bearer <token>" \
     http://localhost:3000/v1/automation/rules/{ruleId}/runs

3. Check worker logs:

   docker compose logs worker | grep automation

Common causes:

• Rule not activated
• Match conditions not met
• Worker service not running
• Rule in draft/sandbox state

Diagnostic Commands

Check Service Status

# All services
docker compose ps

# Specific service
docker compose ps api

View Logs

# All services
docker compose logs -f

# Specific service
docker compose logs -f api

# Last 100 lines
docker compose logs --tail=100 api

Test API Connectivity

# Health check
curl http://localhost:3000/v1/health/live

# With authentication
curl -H "Authorization: Bearer <token>" \
  http://localhost:3000/v1/health/ready

Database Diagnostics

# Connect to database
docker compose exec db psql -U terragnos -d terragnos

# Check table sizes
SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

# Check active connections
SELECT count(*) FROM pg_stat_activity;

License Diagnostics

# Check license status
curl -H "Authorization: Bearer <admin-token>" \
  http://localhost:3000/v1/license

# Check license in database
docker compose exec db psql -U terragnos -d terragnos -c \
  "SELECT * FROM system_licenses ORDER BY created_at DESC LIMIT 1;"

Performance Issues

Slow API Responses

Diagnostic steps:

1. Check database query performance:

   SELECT query, mean_exec_time, calls
   FROM pg_stat_statements
   ORDER BY mean_exec_time DESC
   LIMIT 10;

2. Check for missing indexes:

   SELECT schemaname, tablename, attname, n_distinct
   FROM pg_stats
   WHERE schemaname = 'public'
   AND n_distinct > 100;

3. Monitor API metrics:

   curl http://localhost:3000/metrics | grep http_request_duration

Solutions:

• Add database indexes
• Optimize queries
• Increase database resources
• Enable caching (Redis)

High Memory Usage

Diagnostic steps:

1. Check container memory:

   docker stats

2. Review log levels (reduce verbosity if needed)

3. Check for memory leaks in application logs

Solutions:

• Increase container memory limits
• Reduce log verbosity
• Restart services periodically
• Optimize query patterns

Getting Help

When reporting issues, include:

1. Service logs – Relevant log entries
2. Configuration – Environment variables (sanitized)
3. Error messages – Exact error text
4. Steps to reproduce – What actions led to the issue
5. Version information – Container image tags
6. Health check status – Results from /v1/health/ready

Contact Terragnos support with this information for assistance.