Research for ops-jrz1-mh2: how dev users collaborate on git.clarun.xyz
Covers:
- Account setup and SSH access
- Shared repo vs fork+PR models
- Trunk-based workflow
- Troubleshooting common issues
- Created musiclink repo on Forgejo
- Added dan's devserver SSH key to Forgejo
- Switched musiclink flake input from local path to git+ssh
- Updated musiclink testing room config in modules/musiclink.nix
- Add backup-b2-failed oneshot for OnFailure notification
- Add onFailure handler to both backup-b2 and backup-b2-check
- Add network-online.target dependency to backup-b2-check
- Add TimeoutStartSec (2h for backup, 1h for check)
Found via ops-review lenses.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Tested restore of:
- PostgreSQL dumps (forgejo: 112 tables, mautrix_slack: 32 tables)
- Forgejo repositories
- User home directories
Also updated known gaps status (sops key, PostgreSQL pin fixed).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Prevents automatic upgrade to PostgreSQL 16 when upgrading NixOS.
This allows a safer two-step approach: upgrade NixOS first, then
pg_upgrade later.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Secrets now encrypted to three recipients:
- vultr_vps: server SSH host key (primary)
- admin: workstation key (local editing)
- recovery: offline key at ~/.config/sops/age/recovery.key
If server dies and admin key unavailable, recovery key can
still decrypt secrets to bootstrap restore.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Closes r177. Critical DR gap - user home directories and ACME
certificates were not being backed up.
Excludes common caches that can be rebuilt:
- .cache, .npm/_cacache, .bun/install/cache
- node_modules, .nix-profile, .nix-defexpr
- Trash
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Documents restore procedures for full server loss, partial restore,
and user data recovery scenarios. Includes verification checklists,
time estimates, and break-glass quick reference.
Also documents known gaps (home dirs, ACME, RocksDB consistency)
that need fixing before the runbook is production-ready.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Allows dev users to use nix develop, nix build, etc.
Previously blocked by daemon access restrictions.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Generate server-side SSH keypair for git access from server
- Upload both laptop key and server key to Forgejo
- Add mandatory key revocation in dev-remove.sh
- Fix: use forgejo@ instead of git@ for SSH URLs
- Keys named username-laptop and username-devserver
- Key comment includes DO-NOT-REUSE warning
Closes ops-jrz1-rfx
Write credentials to ~/.forgejo-credentials (JSON, mode 600) when
creating new Forgejo users. Onboarding message points to file
instead of showing password in terminal output.
Addresses ops-jrz1-ofw.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace openssl rand with /dev/urandom (openssl not in NixOS path)
- Update forgejo-api-token with admin scope for user provisioning
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add programs.ssh.knownHosts for git.clarun.xyz (prevents SSH prompts)
- Expose forgejo-api-token via sops-nix for provisioning
- dev-add.sh: Create Forgejo account + upload SSH key via API
- dev-add.sh: Set up .gitconfig with user.name/email
- dev-remove.sh: Print warning to manually suspend Forgejo account
Addresses ops-jrz1-qts.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
