title: Upgrade NixOS from 24.05 to 24.11

description: Running NixOS 24.05.20241230 (Uakari). Current stable is 24.11. May be missing security patches. Low priority as no known critical CVEs, but should plan upgrade.

status: open

issue_type: task

author: dan

text:
Analysis Findings:
1. Version Mismatch: Local flake.nix is pinned to 'nixos-24.05', but the dev environment reports '25.11' (Unstable), indicating state divergence.
2. Upstream Bugs: Blocking issues in mautrix-slack (ops-jrz1-blh) and maubot (sync failure) are present in the current unstable revision (2025-12-02).
3. Recommendation: Upgrade platform to NixOS 24.11 (Stable) to align environment, ensure stability, and pull fresh upstream fixes.


title: Upgrade mautrix-slack to v25.11

description: Upgrade is just flake update + deploy. Current deployed: v0.2.3+dev.unknown (Oct 13). Flake lock: v25.10 (Oct 22). Latest nixpkgs-unstable: v25.11. Run: nix flake update nixpkgs-unstable && deploy. May fix edit panic (ops-jrz1-qxr).

status: closed


title: Direct Slack bot path for learners

description:
Alternative path: learners write Python bots using slack-bolt, connect directly to Slack via Socket Mode. No Matrix, no bridge.

## Architecture
```
learner code → slack-bolt → Socket Mode WebSocket → Slack API
```

## Status

**Done:**
- [x] /etc/slack-learner.env with shared tokens (xoxb-, xapp-)
- [x] learners group for access control (dantest is member)
- [x] learner-add.sh adds users to group, sources env in .bashrc
- [x] Design doc: docs/learner-slack-direct.md

**Not Done:**
- [ ] Starter template (~/slack-bot-template/)
- [ ] Process management (systemd user services or supervisor)
- [ ] #learner-sandbox channel in Slack
- [ ] End-to-end test with real learner

## Tradeoffs vs Maubot/Matrix (ops-jrz1-2pm)
- Faster feedback (direct to Slack)
- Excellent slack-bolt docs
- But: shared bot identity, manual process management

## Ready to Use NOW
Works today with terminal editors (vim/nano):
```bash
ssh alice@ops-jrz1
pip install slack-bolt
python bot.py  # responds in Slack
```

VS Code Remote-SSH needs nix-ld deployed first.

issue_type: epic

created_by: dan


title: Remote dev environment for learners

description:
Set up dev environments for learners to build maubot plugins (Matrix bots that can bridge to Slack).

## Approach
VS Code Remote-SSH + shared maubot + per-user Unix accounts

## Architecture
```
learner code → maubot → Matrix → mautrix-slack bridge → Slack
```

## Status

**Done:**
- [x] learner-add.sh / learner-remove.sh scripts
- [x] Hello-world plugin template (templates/plugin-skeleton/)
- [x] Test user `dantest` created with ~/plugins/hello-dantest/
- [x] Maubot running and healthy

**Not Done:**
- [ ] nix-ld for VS Code Remote-SSH (config added, not deployed)
- [ ] Test full VS Code Remote-SSH flow
- [ ] Test Claude Code extension over Remote-SSH
- [ ] #learners-sandbox Matrix room
- [ ] Onboarding doc polish

## Tradeoffs vs Direct Slack (ops-jrz1-2bu)
- Slower feedback (bridge hop)
- Sparse maubot docs
- But: managed process lifecycle, per-bot identity

## Docs
- docs/learner-onboarding.md
- docs/learner-admin.md


title: Research: Learner deployment pipeline

description:
How does learner code get to "prod" (running services)?

## Current context
- Maubot: Upload .mbp via web UI
- Slack bots: Manual `python bot.py` or systemd user service

## Questions
1. Can learners run persistent services? (`systemctl --user`)
2. Should they have access to maubot admin UI?
3. Git-based deploy? Push to trigger reload?
4. Who can restart what?

## Options
- **Manual only** - Learner runs in foreground/tmux
- **User systemd** - `systemctl --user enable mybot`
- **Supervised** - Central supervisor manages learner procs
- **GitOps** - Push to deploy (complex)

## Security considerations
- What if learner bot crashes in loop?
- Resource limits on user services?
- Can learner affect other learners' services?


title: Research: Agentic coder sandboxing

description:
When Claude Code runs on server, what can it do? Should we limit it?

## Current state
- No sandbox
- Claude has full user privileges
- Can run any command user can run

## Risks
- `rm -rf ~` (accidental or hallucinated)
- Network exfiltration
- Resource exhaustion (fork bomb, disk fill)
- Credential theft from env/files

## Options
1. **Trust the agent** - User's problem if Claude breaks things
2. **Command allowlist** - Only approved commands
3. **Container sandbox** - Run agent in container
4. **Snapshot/rollback** - Easy recovery if things break
5. **Audit logging** - At least know what happened

## Questions
- What do other agentic coding setups do?
- Is this overkill for a learning environment?
- Does Claude Code have built-in safety?


title: Persist opencode state/cache across restarts

description: opencode may store index/cache in ~/.cache or other dirs not covered by current bind mounts. AI context could be lost on container restart. Verify and add mounts.

close_reason: Parent epic cancelled - browser-based dev approach abandoned

type: parent-child

created_by: daemon


title: Deploy and test single-user instance (Phase 1)

description: Deploy one container for testing. Validate: WebSocket, extensions, terminal, opencode, memory usage. Access via SSH tunnel initially.

type: blocks


title: Test: VS Code extension execution location

description:
When using VS Code Remote-SSH, does Claude Code extension run locally or on server?

## Why it matters
- If local: needs to proxy commands to server, may have latency/sync issues
- If remote: has direct access, simpler model, but needs install on server

## Test plan
1. Connect VS Code Remote-SSH to jrz1 as dantest
2. Install Claude Code extension
3. Observe where extension runs (check processes on both sides)
4. Run a simple Claude command, see where execution happens
5. Check if extension needs API key on server or uses local

## Expected outcome
Document actual behavior for onboarding docs.


title: Browser-based dev environment with opencode

description:
Epic: Provide VS Code in browser via code-server with opencode AI integration.

Key decisions:
- code-server in Podman containers (rootless)
- opencode CLI + VS Code extension pre-installed
- Subdomain routing (dan.code.clarun.xyz)
- Custom container image
- Target users: non-programmers, testers, learners

Design doc: specs/004-browser-dev-environment/design.md

Migrated from ops-jrz1-ndl

close_reason: Won't fix - non-MS browser-based VSCode solutions have reliability/compatibility issues


title: Add maubot SDK and deploy script to container image

description:
Container image needs:
- Python 3.11 + maubot SDK
- deploy.sh script (zip → .mbp → curl to maubot API)
- maubot API reachable from container (host network or port forward)

Part of learner onboarding for bot development.


title: Matrix/Slack identity mismatch: dan vs vlad

description: Matrix user @dan:clarun.xyz is linked to Slack user 'vlad'. Messages appear as vlad in Slack but dan in Element. Cosmetic confusion. Options: rename Matrix display name, or re-login bridge with different Slack account.


title: Write onboarding documentation

description: Critical for non-programmers. Cover: login, opencode usage, Git setup (PAT workflow), resource limits, security hygiene. Keep concise.

close_reason: Browser-based dev environment cancelled


title: Smoke test Matrix server (conduwuit)

description: Verify Matrix homeserver is healthy: check /_matrix/client/versions endpoint, test registration, verify federation status (disabled). Quick health check after deployments.


title: Smoke test Maubot service

description: Verify Maubot is healthy: check management UI accessible via SSH tunnel, verify bot instances running, test plugin functionality. Quick health check after deployments.


title: Set up programmatic QA test user for bridge testing


title: Create NixOS module for code-server containers

description: Module to manage per-user Podman containers, nginx routing, secrets. Use virtualisation.oci-containers. Generate systemd units.


title: AI cost/rate limiting per user

description: One user could drain API credits with runaway script. Need rate limiting per user, either via proxy middleware or opencode config. Track usage.


title: Git credential strategy for non-programmers

description: Non-programmers can't manage SSH keys. Pre-configure git-credential-store or provide simple PAT workflow with docs. Store in persistent home with 600 perms.


title: Implement backup strategy for VPS

description: No backups configured. Critical data: Matrix DB (622M), PostgreSQL (161M), Forgejo (2.5M), maubot (320K). No recovery path if disk fails. Need automated backups with off-site storage.


title: Upgrade VPS RAM for dev environments

description: Current: 2GB. Need 4-8GB for multiple code-server containers. Coordinate with Vultr, plan maintenance window.


title: Research: System packages for learner accounts

description:
How do learners get access to toolchains (Go, Node, Rust, etc.)?

## Options
1. **Global install** - Add to `environment.systemPackages`
2. **Per-user nix profile** - `nix profile install nixpkgs#go`
3. **Devshells** - Project-specific `flake.nix` with `nix develop`
4. **direnv + nix** - Auto-activate on cd

## Questions
- What's the NixOS-idiomatic way?
- How much do we want learners to manage themselves?
- Should we provide a "batteries included" default?

## Current state
- python3, uv installed globally
- No Go, Node, Rust


title: Configure wildcard DNS and ACME cert

description: Set up *.code.clarun.xyz DNS record and wildcard SSL cert via ACME. Depends on subdomain routing decision (kg0).


title: Rename sna-instagram-bot to something memorable


title: Enable nix-ld for VS Code Remote SSH

description:
VS Code Remote-SSH requires nix-ld to run the VS Code Server on NixOS.

## Status
- [x] Config added to hosts/ops-jrz1.nix
- [x] nix flake check passed
- [ ] Deploy to server

## Config Added
```nix
programs.nix-ld.enable = true;
programs.nix-ld.libraries = with pkgs; [
  stdenv.cc.cc.lib
  zlib
  openssl
];
```

## Blocks
- VS Code Remote-SSH (won't work without this)
- Testing Claude Code extension over Remote-SSH
- JetBrains Gateway (same issue)

## Deploy Command
```bash
nixos-rebuild switch --flake .#ops-jrz1 --target-host root@ops-jrz1 --build-host localhost
```

close_reason: Deployed - nix-ld active at /lib64/ld-linux-x86-64.so.2


title: Research: Resource limits and quotas

description:
Should we limit CPU/memory/disk per learner?

## Current state
- No limits configured
- Single VPS shared by all users
- 2GB RAM, 1 vCPU (Vultr $6 tier?)

## Options
1. **No limits** - Trust learners, monitor manually
2. **Systemd slices** - cgroups for user sessions
3. **Disk quotas** - Limit ~/
4. **ulimits** - Process limits

## Questions
- What resources does a typical dev session use?
- What about `go build` or `npm install`?
- Is this premature optimization?


title: Add disk quotas for user workspaces

description: User could fill host disk via /var/lib/vscode/<user>/. Add per-directory quotas or monitoring/alerting on disk usage.


title: mautrix-slack edit panic persists in v25.11

description:
mautrix-slack panic on rapid message edits (race condition)

**Root cause**: Edit event arrives before original message is stored in DB. ConvertEdit accesses nil metadata.

**Location**: handleslack.go:575 - has TODO comment: 'this can panic?'

**Reproduction**: Edit a Slack message within ~1 second of sending

**Upstream status**:
- v25.11 is latest (we're on it)
- Known to devs (TODO in code)
- No open issue filed yet

**Stack trace**:
go.mau.fi/mautrix-slack/pkg/connector.(*SlackMessage).ConvertEdit
  handleslack.go:575
maunium.net/go/mautrix/bridgev2.(*Portal).handleRemoteEdit
  portal.go:2838

issue_type: bug

close_reason: Duplicate of ops-jrz1-f15 which has fix ready

text: Confirmed panic exists in nixpkgs-unstable from 2025-12-02. Fix will be addressed via platform upgrade (see ops-jrz1-00e).


title: Build custom code-server container image

description: Dockerfile with: code-server, opencode CLI, opencode VS Code extension (Open VSX), Python, Node, Git. Push to registry or build locally.


title: Port forwarding strategy for user apps

description: When user runs app on localhost:3000, how do they view it? code-server has /proxy/<port> but URL is confusing for learners. Need clear UX or docs.


title: Increase container RAM limits (2GB too tight)

description: 2GB hard limit will OOM with code-server + opencode + LSP + user app. Gemini/GPT recommend 3-4GB per container or add swap. Need to size server appropriately.


title: Container isolation: maubot API access only

description:
Security design for learner containers:

**Container CAN access**:
- maubot API (:29316) for plugin deploy
- Matrix rooms via bot (through maubot)
- Slack via bridge (through Matrix)

**Container CANNOT access**:
- Host filesystem
- Other containers
- PostgreSQL directly
- Matrix homeserver directly
- sops secrets

Implementation: Podman network config, no --privileged, limited port exposure.

close_reason: Depends on cancelled browser-dev epic (ops-jrz1-3so)


title: Maubot plugin dev workflow for learners

description:
Design frictionless dev workflow for Python/Go learners building maubot plugins.

**Requirements**:
- No SSH tunnel setup for learners
- Fast feedback loop (edit → see bot respond)
- Circuit breakers (allowed_rooms, rate limits)
- Test channel: #vlads-pad (Slack) ↔ Matrix

**Options being considered**:
1. Git-push deploy: push to repo → CI builds .mbp → deploys to maubot
2. Code-server containers: browser IDE on VPS, deploy script talks to maubot locally
3. Hybrid: code-server + git workflow

**Related**: ops-jrz1-3so (browser-dev-environment epic)

issue_type: feature


title: mautrix-slack ConvertEdit panic on message edits

description:
mautrix-slack v25.11 panics with nil pointer dereference at handleslack.go:575 when processing Slack message_changed events.

FIX READY at /tmp/mautrix-slack-fix (branch: fix-convert-edit-nil-panic)

Three changes needed in pkg/connector/handleslack.go:
1. Line 181: Add 'evt.SubMessage != nil &&' before evt.SubMessage.SubType
2. Lines 573-578: Add nil checks for SubMessage and SubMessage.Edited at start of ConvertEdit()
3. Line 609: Add 's.Data.SubMessage != nil &&' before accessing Timestamp

To submit PR: fork mautrix/slack, push branch, open PR against main.

close_reason: Won't fix for now - fix ready in /tmp/mautrix-slack-fix if needed later


title: Enable fail2ban for SSH brute force protection

description: SSH brute force attempts generate log noise but don't pose security risk (key-only auth). fail2ban would help but is low priority. Deferred pending RFC on SSH log management strategy.


title: Research: User isolation and sandboxing options

description:
Can users see each other's code? What isolation do we need?

## Current state
- Standard Unix permissions
- No containers or VMs
- Users in same `users` group

## Options to evaluate
1. **Unix perms only** - `chmod 700 ~` (current)
2. **Separate groups** - Per-user primary group
3. **Containers** - systemd-nspawn, podman
4. **VMs** - microVMs per user (heavy)
5. **Firejail/bubblewrap** - Process sandboxing

## Questions
- What's the threat model? (curious learners vs malicious actors)
- How much isolation is worth the complexity?
- Does NixOS offer anything special here?


title: VS Code extension policy (security)

description: Extensions can run arbitrary code. Decide: allow arbitrary installs, or curate/restrict? For non-programmers, pre-install safe set and optionally disable marketplace.


title: Enable mautrix-slack relay mode for bot bridging


title: Instagram bot missing base-config.yaml

description: Plugin was missing base-config.yaml required by maubot Config class. Fixed in commit 4b9481d.


title: Logging and monitoring for dev environments

description: No observability plan. Need: container CPU/mem metrics, nginx logs, disk usage monitoring, alert on repeated 401s or resource exhaustion.


title: Switch to subdomain routing (dan.code.clarun.xyz)

description: Path-based routing (/code/dan/) is fragile. Extensions assume root path, cookies scope incorrectly, PWA breaks. Switch to wildcard subdomains for cleaner isolation.


title: Container reset mechanism (keep workspace)

description: If user breaks their environment, need simple way to wipe container and restore default image while preserving /workspace. Script or admin command.


title: Browser-based dev environment (code-server)

description:
Explore setting up browser-based development:

Options:
- code-server / openvscode-server - VS Code in browser
- ttyd / wetty - terminal in browser
- PWA install to home screen for native app feel

Could combine with Tailscale for secure access without exposing ports.

Ref: ops-dev thin client brainstorm session

notes: Design doc created: specs/004-browser-dev-environment/design.md - covers architecture, tech choices, resource planning, security model, rollout phases


title: RFC: SSH log noise reduction strategy

description: Research showed 99.8% of SSH logs are scanner noise (9000 failed attempts/day). Options: (1) Change SSH port - simple, ~99% reduction (2) journald filter - surgical but complex (3) LogLevel ERROR - loses successful login audit trail (4) fail2ban - bans IPs, partial reduction. Orch consensus: Gemini opposed LogLevel ERROR due to losing audit trail, GPT supported. Need RFC to decide approach. See posture review from Dec 2025 session.


title: Slack bot architecture: Matrix-first approach

description:
**Decision**: Use Matrix as primary platform for Slack bot development.

**Architecture**: Bots run as maubot plugins (or Matrix bots), communicate to Slack via mautrix-slack bridge.

**Rationale**:
- Existing infrastructure (maubot deployed, bridge working)
- Single platform to manage
- Bots work with Matrix users too
- Avoid Socket Mode contention (only one xapp- connection allowed)

**Trade-offs accepted**:
- Bridge dependency (edit panic bug exists)
- Extra latency through bridge hop
- Limited to bridged channels

**Alternative considered (Option B - direct Slack API)**:
- Could use xoxb- token for outbound-only (REST)
- Would need new Slack app for full Socket Mode independence
- Deferred for now

**Credentials available**:
- slack-oauth-token (xoxb-) - shareable for REST calls if needed
- slack-app-token (xapp-) - reserved for bridge Socket Mode

**Status**: DECIDED - staying with Matrix-first


title: mautrix-slack message edit panic (upstream bug)

description: Bridge upgraded to v25.11. Need to verify if edit panic is fixed by testing a Slack message edit. Watch logs: journalctl -u mautrix-slack -f | grep -E 'ERR|panic|edit'


title: Security review of running server


title: Remote dev environment security and setup research

description:
Research and test security/setup questions for learner remote dev environments.

## Context
Two dev paths identified:
1. **Server-first**: SSH in, run agentic coders on server
2. **Local VS Code**: Remote-SSH extension, code on server

Both have open questions around sandboxing, system packages, deployment access, and security boundaries.

## Scope
- System package management for learners
- User isolation and sandboxing options
- Agentic coder security (what can Claude do?)
- Deployment pipeline security
- VS Code extension behavior testing
- Resource limits and quotas

## Deliverables
- Answers to open questions (documented)
- Security recommendations
- Implementation plan for chosen approach


title: Design API key provisioning strategy

description: opencode needs API keys (OpenAI, Anthropic). Options: 1) Shared key with proxy + rate limiting, 2) Per-user keys in sops-nix. Need to prevent key exposure and enable usage tracking.


title: Fix maubot admin UI exposed to internet (port 29316)

description: Maubot admin UI on port 29316 is publicly accessible (returns 401 but API surface exposed). Firewall explicitly allows this port. Risk: brute force on admin password, direct exploit of any maubot vulnerabilities. Fix: bind to 127.0.0.1 only, remove from firewall, access via SSH tunnel.


title: Fix maubot health check (failing every 5 min)

description: Health check at /_matrix/maubot/v1/version returns 401 (auth required). Check script doesn't provide auth token. Spamming error logs every 5 minutes.