# Orchestration Review Lens

Review operational infrastructure for **execution ordering, dependencies, and coupling**.

## What to Look For

### Implicit Dependencies
- Scripts assuming prior state without checking
- Services started without verifying dependencies are ready
- Missing `After=`/`Requires=` in systemd units
- Docker Compose without `depends_on` or health-based waiting
- Kubernetes without init containers for prerequisites

### Startup Ordering
- Database migrations run after app starts
- Config files expected before generation step runs
- Secrets not available when service initializes
- Race conditions between parallel starts

### Shutdown Ordering
- Missing `Before=` in systemd for reverse dependency order
- Missing `preStop` hooks in Kubernetes
- Database stops before app finishes flushing
- No drain period before termination

### Circular Dependencies
- Service A requires B, B requires A
- Deadlocks in systemd ordering
- Flake inputs with circular `follows`
- Scripts that call each other in loops

### Unclear Prerequisites
- No documentation of what must run first
- Missing README or runbook for deployment order
- Makefile targets without dependency declarations
- Scripts without pre-flight checks

### Coupling Issues
- Hard-coded hostnames/ports instead of service discovery
- Direct database access from multiple services (shared state)
- File-based coupling (service A writes, B reads)
- Implicit timing assumptions ("service B is always slower")

### NixOS/systemd Specific
- Missing `Wants=` for optional dependencies
- `After=` without corresponding `Requires=` (ordering without guarantee)
- Activation scripts with implicit ordering
- Missing `PartOf=` for lifecycle coupling

### Docker/Kubernetes Specific
- `depends_on` without `condition: service_healthy`
- Missing `restartPolicy` for transient dependency failures
- Init containers without proper failure handling
- Jobs without `ttlSecondsAfterFinished`

### CI/CD Pipelines
- GitHub Actions jobs without `needs:` for dependencies
- Deployment before build/test completion
- Missing artifact upload/download between jobs
- Parallel jobs accessing same resources without locks

### Job Concurrency
- Scheduled jobs without concurrency control (flock, forbidConcurrency)
- Multiple replicas running migrations simultaneously
- Missing leader election for singleton tasks
- Thundering herd on dependency recovery (add backoff/jitter)

## Output Format

```
[ORCHESTRATION] <severity:HIGH|MED|LOW> <file:line>
Issue: <what ordering/dependency is unclear or broken>
Impact: <race condition, startup failure, implicit coupling>
Suggest: <explicit dependency, health check, documentation>
Evidence: <pattern found>
```

## Common Fixes

```ini
# systemd explicit ordering with guarantee
[Unit]
After=postgresql.service
Requires=postgresql.service
Wants=redis.service  # optional dependency
```

```yaml
# Docker Compose health-based dependency
services:
  app:
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
```

```yaml
# Kubernetes init container
initContainers:
  - name: wait-for-db
    image: busybox
    command: ['sh', '-c', 'until nc -z db 5432; do sleep 1; done']
```

```makefile
# Makefile with explicit dependencies
deploy: build test migrate
	./deploy.sh

migrate: db-ready
	./run-migrations.sh
```

```bash
# Script with pre-flight check
#!/bin/bash
set -euo pipefail

# Verify prerequisites
command -v jq >/dev/null || { echo "jq required"; exit 1; }
[ -f /etc/app/config.yaml ] || { echo "Config missing"; exit 1; }
curl -sf http://db:5432/health || { echo "DB not ready"; exit 1; }
```

## False Positive Risks

- Intentionally loose coupling for flexibility
- Services with internal retry logic that handle missing dependencies
- Development environments with simplified ordering
- Stateless services that can start in any order
- Service mesh (Istio/Linkerd) handles sidecar injection automatically
- Event-driven systems designed for eventual consistency
- Init container `nc -z` redundant if app has robust retry logic

## Guidelines

- **HIGH** = race conditions, circular deps, startup failures in production
- **MED** = implicit ordering, missing health checks, undocumented prerequisites
- **LOW** = could be more explicit, minor coupling concerns
- Ask: "What happens if this starts before its dependency? What's the contract?"
- Explicit is better than implicit - document and enforce ordering
- Health checks > timing assumptions
