Operations Guide

Health Checks

The backend exposes /healthz for liveness and /readyz for readiness:

curl http://localhost:8080/healthz
curl http://localhost:8080/readyz

Through nginx (TLS):

curl -k https://localhost/healthz

Logs

# All services
make prod-logs

# Specific service
docker logs -f umoo-backend
docker logs -f umoo-postgres
docker logs -f umoo-nginx

The backend outputs structured JSON logs in production (UMOO_LOG_FORMAT=json).

Monitoring

NATS

NATS monitoring is available at http://localhost:8222:

curl http://localhost:8222/varz    # Server variables
curl http://localhost:8222/connz   # Connection info
curl http://localhost:8222/routez  # Route info
curl http://localhost:8222/subsz   # Subscription info

PostgreSQL

Check connection pool and active queries:

docker exec umoo-postgres psql -U umoo -d umoo -c \
  "SELECT pid, state, query_start, query FROM pg_stat_activity WHERE datname='umoo';"

Redis

docker exec umoo-redis redis-cli -a $REDIS_PASSWORD INFO memory
docker exec umoo-redis redis-cli -a $REDIS_PASSWORD INFO clients

Backup and Restore

PostgreSQL Backup

docker exec umoo-postgres pg_dump -U umoo umoo > backup_$(date +%Y%m%d_%H%M%S).sql

PostgreSQL Restore

docker exec -i umoo-postgres psql -U umoo umoo < backup_20260101_120000.sql

Redis Backup

Redis is configured with AOF persistence. Data is stored in the redis_data volume.

docker exec umoo-redis redis-cli -a $REDIS_PASSWORD BGSAVE
docker cp umoo-redis:/data/dump.rdb ./redis_backup_$(date +%Y%m%d).rdb

Troubleshooting

Service Won't Start

# Check container status
docker ps -a | grep umoo

# Check recent logs
docker logs --tail 50 umoo-backend

# Verify dependencies are healthy
docker inspect --format='{{.State.Health.Status}}' umoo-postgres
docker inspect --format='{{.State.Health.Status}}' umoo-redis
docker inspect --format='{{.State.Health.Status}}' umoo-nats-1

Database Connection Issues

# Test connectivity from backend container
docker exec umoo-backend sh -c 'nc -z postgres 5432 && echo OK || echo FAIL'

# Check PostgreSQL max connections
docker exec umoo-postgres psql -U umoo -c "SHOW max_connections;"
docker exec umoo-postgres psql -U umoo -c "SELECT count(*) FROM pg_stat_activity;"

NATS Cluster Problems

# Check cluster routes
curl -s http://localhost:8222/routez | python3 -m json.tool

# Verify all nodes see each other
for port in 8222; do
  echo "Node on $port:"
  curl -s "http://localhost:$port/routez" | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'  Routes: {d[\"num_routes\"]}')"
done

Rate Limiting

If clients receive HTTP 429 responses, check the rate limit configuration:

• Nginx: limit_req_zone in nginx.conf (default 100r/s per IP)
• Application: Per-tenant rate limiting via Redis (configurable per tenant)

Graceful Shutdown

The backend supports graceful drain. On SIGTERM:

1. Health endpoint returns unhealthy (nginx stops routing)
2. In-flight requests complete (30s timeout)
3. WebSocket connections drain
4. Database and Redis connections close
5. Process exits

# Graceful restart
docker restart umoo-backend

# Or via compose
make prod-down && make prod-up

Resource Monitoring

docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"