# Observability Review Lens

Review operational infrastructure for **visibility, monitoring, and debuggability**.

## What to Look For

### Silent Failures
- Commands with stderr redirected to /dev/null without reason
- Missing exit code checks after critical operations
- catch/except blocks that swallow errors silently
- Cron jobs without output capture or MAILTO
- systemd services without logging (StandardOutput/StandardError)

### Health Checks
- Docker: Missing HEALTHCHECK instruction
- Compose: Missing `healthcheck:` for services with dependencies
- Kubernetes: Missing readiness/liveness probes
- systemd: Missing `ExecStartPre` validation or `Type=notify`
- No startup verification before declaring "ready"

### Logging Quality
- Print statements instead of structured logging
- Missing log levels (debug/info/warn/error)
- Logs without timestamps
- No context in error messages (what failed, with what input?)
- Secrets/PII in log output

### Correlation & Tracing
- Multi-step scripts without operation IDs
- Distributed operations without trace/correlation IDs
- Missing request IDs in API services
- No way to follow a request across components

### Metrics & Alerting
- Long-running services without metrics endpoint
- Missing duration/latency tracking for operations
- No alerting on critical failures
- Batch jobs without success/failure metrics
- Missing SLI/SLO instrumentation
- Scheduled jobs without heartbeat/dead-man's-switch (job didn't run at all)

### Resource Visibility
- No monitoring for disk space, inodes, file descriptors
- Missing memory/OOM visibility (services killed silently)
- Log directories without rotation policies
- No resource limit alerts before exhaustion

### Debugging Capability
- No way to enable verbose/debug mode
- Missing dry-run for complex operations
- No state inspection commands
- Logs that don't include enough context to reproduce issues
- Missing version/commit hash in startup logs (was the deploy successful?)
- No config dump on startup (redacted) to verify active configuration

## Output Format

```
[OBSERVABILITY] <severity:HIGH|MED|LOW> <file:line>
Issue: <what's invisible or silent>
Impact: <debugging difficulty, missed failures>
Suggest: <add logging, health check, metrics>
Evidence: <pattern found>
```

## Common Fixes

```dockerfile
# Docker health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \
  CMD curl -f http://localhost:8080/health || exit 1
```

```yaml
# Compose health check
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s
```

```ini
# systemd logging and notification
[Service]
Type=notify
StandardOutput=journal
StandardError=journal
ExecStartPre=/usr/bin/test -f /etc/myapp/config.yaml
```

```bash
# Script with operation ID
OP_ID="${OP_ID:-$(date +%s)-$$}"
log() { echo "[$(date -Iseconds)] [$OP_ID] $*" >&2; }
log "Starting backup operation"
```

## False Positive Risks

- Intentionally quiet commands in pipelines (intermediate steps)
- Services with external health monitoring (not self-checks)
- Development/test environments where full observability is overhead
- K8s ignores Dockerfile HEALTHCHECK; probes are the real control
- Cron without MAILTO when jobs emit metrics instead
- Utility/sidecar containers that don't need full instrumentation

## Guidelines

- **HIGH** = silent failures in production, no health checks on critical services
- **MED** = missing structured logging, no correlation IDs
- **LOW** = could improve debugging, missing nice-to-have metrics
- Ask: "If this fails at 3 AM, how would we know? How would we debug?"
- Containers should be observable from outside (health, logs, metrics)
- Every operation should be traceable from start to finish
