# Resilience Review Lens

Review operational infrastructure for **runtime fault tolerance and graceful degradation**.

## What to Look For

### Timeouts
- Network calls without timeout (curl, wget, API clients)
- Database connections without connect/query timeout
- HTTP clients with no deadline or infinite timeout
- Missing `TimeoutStartSec`/`TimeoutStopSec` in systemd

### Retries & Backoff
- Retry logic without exponential backoff
- Missing jitter (thundering herd on recovery)
- Infinite retry loops without circuit breaker
- No max retry limit

### Circuit Breakers
- External API calls without failure threshold
- Database connections that retry forever on outage
- Missing fallback behavior when dependency unavailable
- No degraded mode for non-critical features

### Graceful Shutdown
- No SIGTERM handler (abrupt termination)
- Missing drain period for in-flight requests
- Database connections not closed on shutdown
- systemd `KillMode=control-group` without `ExecStop`
- Missing `stop_grace_period` in Docker Compose

### Resource Limits
- systemd: Missing `MemoryMax`, `CPUQuota`, `TasksMax`
- Docker: Missing `mem_limit`, `cpus`, `pids_limit`
- Kubernetes: Missing resource requests/limits
- No `ulimit` for file descriptors in high-connection services
- Missing `LimitNOFILE` in systemd for network services

### Connection Management
- No connection pooling for databases
- Missing connection limits (max connections)
- No idle timeout for connection pools
- Connections held across retries (stale connections)

### Rate Limiting
- No rate limiting on API endpoints
- Missing backpressure handling for queues
- Unbounded work queues that grow under load

### Health Checks & Self-Healing
- Missing liveness probes (deadlocked process not restarted)
- Missing readiness probes (traffic sent before initialization)
- Aggressive liveness probes that fail on dependency outages (restart loops)
- Missing `WatchdogSec` in systemd for self-healing
- No startup probe/warmup period before traffic

### DNS & Network
- DNS caching forever (fails on failover/IP changes)
- No DNS resolution timeout
- Missing TCP keepalives for detecting dead connections

### Storage & Logging
- Unbounded logging filling disk (missing log rotation)
- Docker without `max-size` log option
- Missing disk space checks before write-heavy operations

## Output Format

```
[RESILIENCE] <severity:HIGH|MED|LOW> <file:line>
Issue: <what fails under stress or partial outage>
Impact: <cascade failure, resource exhaustion, hung process>
Suggest: <timeout, backoff, circuit breaker, limit>
Evidence: <pattern found>
```

## Common Fixes

```bash
# curl with timeout
curl --connect-timeout 5 --max-time 30 "$url"

# wget with timeout
wget --timeout=30 --tries=3 "$url"
```

```ini
# systemd resource limits and timeouts
[Service]
TimeoutStartSec=30
TimeoutStopSec=30
MemoryMax=512M
CPUQuota=50%
TasksMax=100
LimitNOFILE=65535
```

```yaml
# Docker Compose limits
services:
  app:
    mem_limit: 512m
    cpus: 0.5
    pids_limit: 100
    stop_grace_period: 30s
```

```python
# Python retry with backoff
import backoff

@backoff.on_exception(
    backoff.expo,
    requests.exceptions.RequestException,
    max_tries=5,
    jitter=backoff.full_jitter
)
def call_api():
    return requests.get(url, timeout=30)
```

## False Positive Risks

- Local-only operations don't need network timeouts
- Batch jobs may intentionally run without time limits
- Development environments don't need production resource limits
- Some services are designed to wait indefinitely (message queues)
- WebSockets/SSE/long-polling have different timeout semantics
- Service mesh (Istio/Envoy) may handle retries at infrastructure layer
- systemd default stop behavior may be sufficient (no ExecStop needed)

## Guidelines

- **HIGH** = no timeout on external calls, missing graceful shutdown in production
- **MED** = no backoff/jitter, missing resource limits, no connection pooling
- **LOW** = could be more defensive, missing nice-to-have limits
- Ask: "What happens when the database is slow? When the API is down?"
- Every external call needs a timeout - no exceptions
- **Retry safety**: Only suggest retries for read-only or known-idempotent operations
- Does NOT own: re-run safety (idempotency), change safety (blast-radius)
