ops-jrz1/docs/worklogs/2025-10-26-slack-bridge-deployment-complete.org

#+TITLE: mautrix-slack Bridge Deployment - Socket Mode Integration Complete
#+DATE: 2025-10-26
#+KEYWORDS: mautrix-slack, matrix, slack, bridge, conduwuit, socket-mode, nixos, appservice
#+COMMITS: 0
#+COMPRESSION_STATUS: uncompressed

* Session Summary
** Date: 2025-10-26 (Day 4 of feature 002-slack-bridge-integration)
** Focus Area: Deploy and debug mautrix-slack bridge with Socket Mode, troubleshoot conduwuit appservice registration issues

* Accomplishments
- [X] Successfully deployed mautrix-slack bridge with Socket Mode integration
- [X] Fixed critical IPv4/IPv6 networking issues preventing Matrix homeserver connectivity
- [X] Resolved conduwuit appservice registration and authentication issues by wiping stale database
- [X] Configured Slack app with complete OAuth scopes and Event Subscriptions
- [X] Achieved bidirectional message flow: Matrix ↔ Slack working
- [X] Synced ~50 Slack channels to Matrix rooms automatically
- [X] Updated dev-services.nix with IPv4 fixes for nginx and bridge configuration
- [X] Documented complete Slack app configuration requirements for Socket Mode bridges
- [ ] Commit changes to git (pending)
- [ ] Write comprehensive feature documentation (pending)

* Key Decisions

** Decision 1: Use Socket Mode instead of traditional webhook-based bridge
- Context: mautrix-slack supports two connection modes: webhooks (requiring public endpoint) or Socket Mode (WebSocket connection)
- Options considered:
  1. Webhooks - Traditional approach, requires public endpoint, more complex firewall setup
  2. Socket Mode - WebSocket-based, no public endpoint needed, simpler for self-hosted
- Rationale: Socket Mode eliminates need for public endpoint, simplifies security model, perfect for self-hosted VPS
- Impact: Requires app-level token (xapp-) in addition to bot token (xoxb-), different event delivery model

** Decision 2: Wipe conduwuit database instead of continuing to debug stale state
- Context: After multiple attempts to register appservice, conduwuit consistently rejected as_token with M_UNKNOWN_TOKEN
- Options considered:
  1. Continue debugging conduwuit's internal appservice storage mechanisms
  2. Deep dive into RocksDB to manually inspect/fix registration state
  3. Wipe database and start fresh
- Rationale: We had migrated through multiple conduwuit versions (rc.6 → rc.8), database schema v17 → v18, and multiple failed appservice registration attempts. Fresh database would eliminate any stale state.
- Impact: Lost all existing Matrix rooms and user accounts, but this was acceptable since we were still in setup phase with no production data. **This was the breakthrough that solved the as_token rejection issue.**

** Decision 3: Use explicit IPv4 addresses (127.0.0.1) instead of localhost
- Context: Multiple connection failures between services using "localhost"
- Options considered:
  1. Continue using "localhost" and investigate DNS/resolver configuration
  2. Switch to explicit IPv4 addresses (127.0.0.1)
  3. Configure dual-stack IPv4/IPv6 properly
- Rationale: "localhost" was resolving to IPv6 [::1] but services only listening on IPv4 127.0.0.1, causing connection refused errors
- Impact: Required changes in three places:
  - nginx proxy_pass directives (Matrix and Forgejo)
  - mautrix-slack bridge homeserver URL
  - Simplified troubleshooting by removing DNS resolution layer

** Decision 4: Enable debug logging on conduwuit temporarily
- Context: Needed visibility into why appservice authentication was failing
- Rationale: Default "info" level wasn't showing appservice-related activity
- Impact: Changed log level to "debug" in dev-services.nix, though ultimately the fresh database solved the issue before debug logs revealed anything. Left at debug level for now to aid in future troubleshooting.

* Problems & Solutions

| Problem | Solution | Learning |
|---------|----------|----------|
| Bridge crashes with "as_token was not accepted" despite appservice being registered | Wiped conduwuit database completely (rm -rf /var/lib/matrix-continuwuity/db) and re-registered appservice on fresh database | Conduwuit's appservice registration storage can get into stale/corrupted state during database migrations or after multiple registration attempts. Fresh database with proper registration sequence works reliably. Always consider database wipe as valid troubleshooting step in pre-production. |
| nginx returns "connect() failed (111: Connection refused) while connecting to upstream, upstream: http://[::1]:8008" | Changed all nginx proxy_pass directives from "http://localhost:8008" to "http://127.0.0.1:8008" | localhost can resolve to either IPv4 or IPv6 depending on /etc/hosts and resolver configuration. Services explicitly binding to 127.0.0.1 won't accept connections to [::1]. Be explicit about IP version in service-to-service communication. |
| Bridge crashes with "dial tcp [::1]:8008: connect: connection refused" | Changed mautrix-slack homeserverUrl from "http://localhost:8008" to "http://127.0.0.1:8008" in dev-services.nix | Same IPv4/IPv6 issue affected bridge-to-homeserver communication. This was the second location that needed fixing after nginx. |
| Bridge generates registration file with random sender_localpart instead of "slackbot" | mautrix-slack's registration generator doesn't respect bot.username config when generating YAML. Had to manually edit registration file after generation to change sender_localpart from random string to "slackbot" | The NixOS module generates config correctly (bot.username = "slackbot"), but the bridge's -g flag to generate registration uses a random sender_localpart. Must manually fix after generation before registering with homeserver. |
| Matrix homeserver not routing messages to bridge bot despite appservice registration | Restarted conduwuit after registering appservice. Initial registration while homeserver already running didn't take effect | Conduwuit documentation mentions "if it doesn't work, restarting while the appservice is running could help". This is required - appservice registration via admin room doesn't hot-reload without restart. |
| Slack messages not flowing to Matrix despite successful authentication | Event Subscriptions not configured in Slack app settings | Socket Mode still requires Event Subscriptions to tell Slack which events to push through the WebSocket. Added: message.channels, message.groups, message.im, message.mpim, reaction_added, reaction_removed |
| Build/deploy failures with "Broken pipe" during package transfer to VPS | Used rsync + VPS-side build pattern instead of --target-host: rsync project to VPS, then ssh and build on VPS directly | Large Nix packages (100MB+ like glibc) timeout when transferring over SSH during nixos-rebuild --target-host. Building on VPS allows it to download from cache.nixos.org directly with better bandwidth. |

* Technical Details

** Code Changes
- Total files modified: 19
- Key files changed:
  - `modules/dev-services.nix` - Critical IPv4 fixes and debug logging:
    - Line 119: Changed log level from "info" to "debug"
    - Lines 247, 254: Changed nginx proxy_pass from "localhost" to "127.0.0.1" for Matrix endpoints
    - Lines 281, 288, 295: Changed nginx proxy_pass from "localhost" to "127.0.0.1" for Forgejo endpoints
    - Line 207: Changed mautrix-slack homeserverUrl from "localhost" to "127.0.0.1"
  - `.specify/*` - Spec-kit framework updates (unrelated to this feature)
  - `CLAUDE.md` - Updated with Slack bridge deployment commands and patterns

- Manual changes on VPS (not yet in code):
  - `/var/lib/matrix-appservices/mautrix_slack_registration.yaml` - Manually edited sender_localpart to "slackbot"
  - Wiped `/var/lib/matrix-continuwuity/db/` - Fresh database with proper schema v18

** Commands Used

Deployment pattern (successful after multiple iterations):
```bash
# Sync project to VPS
rsync -avz --exclude '.git' --exclude 'result' --exclude 'ops-jrz1-vm.qcow2' \
  /home/dan/proj/ops-jrz1/ root@45.77.205.49:/root/ops-jrz1/

# Build and activate on VPS
ssh root@45.77.205.49 'cd /root/ops-jrz1 && nixos-rebuild switch --flake .#ops-jrz1'
```

Database wipe and fresh start:
```bash
# Stop services
ssh root@45.77.205.49 'systemctl stop matrix-continuwuity mautrix-slack'

# Wipe Matrix database
ssh root@45.77.205.49 'rm -rf /var/lib/matrix-continuwuity/db'

# Wipe bridge database
ssh root@45.77.205.49 'sudo -u postgres psql -c "DROP DATABASE mautrix_slack;"'
ssh root@45.77.205.49 'sudo -u postgres psql -c "CREATE DATABASE mautrix_slack OWNER mautrix_slack;"'

# Restart Matrix (creates fresh DB)
ssh root@45.77.205.49 'systemctl start matrix-continuwuity'

# Start bridge
ssh root@45.77.205.49 'systemctl start mautrix-slack'
```

Appservice registration in Matrix admin room:
```
!admin appservices register
```yaml
id: slack
url: http://127.0.0.1:29319
as_token: uso7hrUjAKs665c7qHRYtyx7cfDBxLpu4nX4UpXVlXzQ9sQD4a6y0KLglbNiCt2H
hs_token: hqAUGMcllDmP6kRZefdhNWOQMBUrqh5aNXp0nZiu4TAL9a7WhiAKwke634ggqIvw
sender_localpart: slackbot
rate_limited: false
namespaces:
    users:
        - regex: ^@slackbot:clarun\.xyz$
          exclusive: true
        - regex: ^@slack_.*:clarun\.xyz$
          exclusive: true
de.sorunome.msc2409.push_ephemeral: true
receive_ephemeral: true
```
```

Verification commands:
```bash
# Check Matrix homeserver logs
ssh root@45.77.205.49 'journalctl -u matrix-continuwuity --since "5 minutes ago" --no-pager'

# Check bridge status and logs
ssh root@45.77.205.49 'systemctl status mautrix-slack'
ssh root@45.77.205.49 'journalctl -u mautrix-slack --since "5 minutes ago" --no-pager'

# Test Matrix API endpoints
ssh root@45.77.205.49 'curl -s http://127.0.0.1:8008/_matrix/client/versions | jq .'
ssh root@45.77.205.49 'curl -s http://127.0.0.1:8008/_matrix/client/v3/profile/@slackbot:clarun.xyz'

# Check nginx errors
ssh root@45.77.205.49 'journalctl -u nginx --since "5 minutes ago" --no-pager | grep -i error'

# Test bridge endpoint
ssh root@45.77.205.49 'curl -s http://127.0.0.1:29319/_matrix/app/v1/ping'
```

** Architecture Notes

Matrix-Slack Bridge Architecture with Socket Mode:
```
┌─────────────────────────────────────────────────────────────┐
│ Slack Workspace (chochacho)                                  │
│  - Channels (~50 synced)                                     │
│  - Users, Messages, Reactions, Files                         │
└────────────┬────────────────────────────────────────────────┘
             │
             │ Socket Mode WebSocket
             │ (Slack pushes events)
             │
             ▼
┌─────────────────────────────────────────────────────────────┐
│ mautrix-slack Bridge (Port 29319)                           │
│  - Maintains WebSocket to Slack API                         │
│  - Translates Slack events → Matrix events                  │
│  - Sends Matrix events → Slack via API (bot token)          │
│  - Creates/manages portal rooms (Slack channel ↔ Matrix)    │
└────────────┬────────────────────────────────────────────────┘
             │
             │ Appservice Protocol (HTTP)
             │ as_token / hs_token auth
             │
             ▼
┌─────────────────────────────────────────────────────────────┐
│ conduwuit Matrix Homeserver (Port 8008)                     │
│  - Routes messages to/from bridge                           │
│  - Manages Matrix rooms, users, state                       │
│  - RocksDB backend (schema v18)                             │
└────────────┬────────────────────────────────────────────────┘
             │
             │ Client-Server API (HTTPS via nginx)
             │
             ▼
┌─────────────────────────────────────────────────────────────┐
│ Matrix Clients (Element Desktop)                            │
│  - Users interact with bridged Slack channels               │
│  - Messages appear in Matrix rooms                          │
└─────────────────────────────────────────────────────────────┘
```

Key networking requirements:
- Bridge → Matrix: HTTP to 127.0.0.1:8008 (must be IPv4)
- Matrix → Bridge: HTTP to 127.0.0.1:29319 (appservice callbacks)
- Bridge → Slack: WebSocket outbound (Socket Mode)
- Nginx → Matrix: HTTP to 127.0.0.1:8008 (must be IPv4)
- Clients → Nginx: HTTPS to clarun.xyz:443

Security model:
- Bridge has two secrets: as_token (authenticates to Matrix) and hs_token (Matrix authenticates to bridge)
- Slack has two tokens: xoxb- bot token (API calls) and xapp- app-level token (Socket Mode connection)
- No public endpoints needed for bridge (Socket Mode eliminates webhook requirement)
- All secrets managed via sops-nix, deployed to /run/secrets/

** Slack App Configuration Requirements

For a Socket Mode Matrix bridge, the Slack app needs:

**OAuth & Permissions → Bot Token Scopes:**
```
channels:history     - Read messages in public channels
channels:read        - List/see public channels
channels:join        - Auto-join channels (optional but useful)
chat:write          - Send messages as bot
chat:write.customize - Send with custom name/avatar (recommended)
files:read          - Download file attachments
files:write         - Upload file attachments
groups:history      - Read messages in private channels
groups:read         - List/see private channels
im:history          - Read direct messages
im:read             - List/see DMs
im:write            - Send DMs
mpim:history        - Read group DMs
mpim:read           - List/see group DMs
mpim:write          - Send group DMs
reactions:read      - See emoji reactions
reactions:write     - Add emoji reactions
team:read           - Get workspace info
users:read          - Get user profiles (names, avatars)
```

**Socket Mode:**
- Enable Socket Mode
- Generate app-level token with `connections:write` scope
- Token format: xapp-1-{workspace_id}-{timestamp}-{long_hex_string}

**Event Subscriptions:**
- Enable Event Subscriptions
- Subscribe to bot events:
  - `message.channels` - Messages in public channels
  - `message.groups` - Messages in private channels
  - `message.im` - Direct messages
  - `message.mpim` - Group direct messages
  - `reaction_added` - Reaction added
  - `reaction_removed` - Reaction removed

**Installation:**
- Install to workspace → generates Bot User OAuth Token (xoxb-...)
- Both tokens (xapp- and xoxb-) needed for bridge authentication

* Process and Workflow

** What Worked Well

1. **Systematic troubleshooting approach**: Started with logs, identified specific error messages, isolated to networking/auth issues

2. **Breaking down the problem**: Separated concerns:
   - Is the bridge running? (Yes)
   - Can bridge reach Matrix? (No - IPv6 issue)
   - Can Matrix reach bridge? (After IPv6 fix, yes)
   - Are they authenticating? (No - stale DB state)
   - Is Slack connected? (Yes, after auth)
   - Are events flowing? (Yes, after Event Subscriptions)

3. **Using existing worklogs**: Referenced `docs/worklogs/2025-10-22-deployment-generation-31.md` for successful rsync+VPS-build deployment pattern

4. **Web search for conduwuit specifics**: Found documentation about appservice registration requiring restart

5. **Fresh database as solution**: Rather than continuing to debug complex state issues, starting fresh eliminated all variables

** What Was Challenging

1. **IPv4/IPv6 resolution mystery**: Took multiple iterations to identify that "localhost" was resolving to IPv6 but services were IPv4-only. Required fixing in 3 separate places (nginx Matrix, nginx Forgejo, bridge config).

2. **Conduwuit appservice registration opacity**: Homeserver logged nothing at info or debug level about appservice registration attempts or failures. Made it very difficult to understand what was wrong.

3. **mautrix-slack registration generator behavior**: The bridge's -g flag generates registration with random sender_localpart, ignoring the bot.username from config.yaml. This mismatch between registration and config caused authentication failures. No clear documentation about this.

4. **Multiple failed deployment attempts**: Network timeouts during nixos-rebuild --target-host, requiring switch to rsync pattern. Each failed deploy took 2-5 minutes.

5. **Slack app configuration iteration**: Had to request workspace admin approval multiple times as we discovered additional required scopes and Event Subscriptions. "Boss will fire me from pinball startup" was mentioned. Final comprehensive list of all scopes avoided further approvals.

** False Starts and Dead Ends

1. **Attempted to fix conduwuit by examining source code**: Downloaded conduwuit source, looked for appservice registration storage mechanisms. This was unnecessary - fresh database was simpler solution.

2. **Tried to manually test as_token via curl**: Attempted to validate as_token by calling Matrix API directly. This showed token was rejected, but didn't reveal why. Wasted time on this diagnostic dead end.

3. **Increased log level to debug**: Changed conduwuit log level to "debug" expecting to see appservice activity. Even at debug level, conduwuit logged nothing useful about appservice issues. Fresh database solved it before debug logs helped.

4. **Multiple appservice re-registrations**: Tried unregistering and re-registering appservice multiple times via admin room. This didn't help because the underlying database state was corrupted from schema migrations.

5. **Attempted to use admin API instead of admin room**: Tried curl POST to conduwuit admin endpoints for appservice registration. Got M_UNRECOGNIZED error. Admin room commands were the correct approach.

* Learning and Insights

** Technical Insights

1. **Conduwuit appservice registration is fragile**: Database schema migrations (v17 → v18) or multiple registration attempts can leave appservice state corrupted. Fresh database with clean registration sequence is most reliable. Consider this normal troubleshooting step.

2. **Socket Mode architecture advantages**: No public endpoints needed, simpler security model, WebSocket provides reliable event delivery. Much better for self-hosted bridges than traditional webhooks.

3. **IPv4/IPv6 explicit binding**: Services in systemd that bind to 127.0.0.1 explicitly will NOT accept connections to [::1]. Always use explicit IP addresses in service-to-service communication to avoid resolver ambiguity.

4. **NixOS two-stage configuration pattern**: Bridge module uses two-stage ExecStartPre:
   - Stage 1 (root): Create directories, set ownership
   - Stage 2 (service user): Generate config from template, merge with NixOS options
   This allows declarative config while handling runtime secrets properly.

5. **mautrix bridge registration generation**: Bridges use -g flag to generate registration YAML, but this generator doesn't always respect config values. Manual editing of generated registration may be needed before registering with homeserver.

** Process Insights

1. **Database wipe is valid troubleshooting**: In pre-production, don't hesitate to wipe state and start fresh. It's often faster than debugging complex state corruption issues.

2. **Document all configuration requirements comprehensively**: For external service integrations (like Slack app), gather ALL required scopes/settings upfront to avoid multiple approval cycles. Better to request extra permissions you might not use than to go back for more.

3. **Use explicit IP addresses in config**: Don't rely on "localhost" resolution. Be explicit about IPv4 (127.0.0.1) vs IPv6 (::1) to avoid subtle networking issues.

4. **Test bidirectionally**: When integrating two systems, test both directions independently:
   - Matrix → Slack worked first
   - Slack → Matrix required Event Subscriptions
   Helped isolate which side had the issue.

5. **Preserve working deployment patterns**: The rsync + VPS-build pattern from previous worklog (2025-10-22) saved significant time. Keep successful patterns documented and reuse them.

** Architectural Insights

1. **Appservice protocol simplicity**: Once properly configured, appservice protocol is straightforward: bridge listens on port, homeserver makes HTTP callbacks with hs_token. The complexity is in registration and authentication setup.

2. **Matrix homeserver modularity**: conduwuit's admin room commands provide good runtime management of appservices without requiring config file edits or restarts (though restarts help in some cases).

3. **Bridge portal model**: mautrix bridges use "portals" (Slack channel ↔ Matrix room mappings). The bridge automatically created ~50 Matrix rooms during initial sync, one per Slack channel. This automatic discovery is very powerful.

4. **Event-driven architecture**: Socket Mode pushes Slack events to bridge through WebSocket. Bridge transforms and forwards to Matrix. This is cleaner than polling or webhook callbacks.

5. **Security token model**: Four distinct tokens in play:
   - as_token: Bridge authenticates to Matrix
   - hs_token: Matrix authenticates to bridge
   - xoxb-: Bot token for Slack API calls
   - xapp-: App-level token for Socket Mode
   Each serves specific purpose, all must be configured correctly.

* Context for Future Work

** Open Questions

1. **Double puppeting**: Should we enable double puppeting so Matrix users appear as themselves in Slack rather than through bot? This requires additional configuration and user setup.

2. **Encryption support**: Should bridged rooms support Matrix E2E encryption? mautrix bridges support this but it adds complexity. Current setup works without it.

3. **Historical message backfill**: Should we backfill historical Slack messages into Matrix rooms? Bridge can do this but it's expensive on first sync.

4. **User provisioning**: How should new users be onboarded? Currently manual registration with token. Could automate via SSO or registration API.

5. **Monitoring and health checks**: What metrics should we expose? How do we monitor bridge health in production? Currently only systemd status and logs.

6. **Database backup strategy**: conduwuit uses RocksDB, mautrix uses PostgreSQL. What's the backup/restore process? How do we handle database upgrades?

** Next Steps

1. **Commit configuration changes**:
   - modules/dev-services.nix (IPv4 fixes and debug logging)
   - Update CLAUDE.md with deployment commands
   - Create git commit documenting the changes

2. **Test additional bridge features**:
   - File attachments (upload/download)
   - Emoji reactions (bidirectional)
   - Thread replies (if supported)
   - User profile sync (names, avatars)
   - Edit/delete message sync

3. **Update tasks.md**:
   - Mark infrastructure tasks (T008-T010) as complete
   - Update status of authentication tasks
   - Document remaining features to test

4. **Write deployment runbook**: Create quickstart.md with:
   - Fresh deployment from scratch
   - Common troubleshooting steps
   - Slack app configuration checklist
   - Recovery procedures

5. **Consider turning debug logging back to info**: Currently at "debug" level which may be too verbose for production

6. **Test failure scenarios**:
   - What happens if Slack WebSocket disconnects?
   - What happens if Matrix homeserver restarts?
   - How does bridge handle rate limits?

7. **Performance testing**:
   - High-volume channel message handling
   - Large file attachment transfer
   - Many concurrent users

** Related Work

- Previous worklog: `docs/worklogs/2025-10-22-forgejo-repository-setup.org` - Not directly related but shows project progression
- Previous worklog: `docs/worklogs/2025-10-22-deployment-generation-31.md` - Contains successful rsync+VPS-build deployment pattern that we reused
- Spec directory: `specs/002-slack-bridge-integration/` - Contains:
  - spec.md - Feature specification
  - plan.md - Implementation plan
  - research.md - Socket Mode research
  - tasks.md - Task breakdown (67 tasks across 7 phases)

** External Documentation Consulted

1. **mautrix-slack documentation**:
   - https://docs.mau.fi/bridges/general/troubleshooting.html - Bridge troubleshooting guide
   - https://docs.mau.fi/faq/as-token - Appservice token authentication

2. **conduwuit documentation**:
   - https://conduwuit.puppyirl.gay/appservices.html - Appservice registration via admin room
   - Found through web search when trying to understand registration process

3. **Slack API documentation**:
   - Socket Mode setup and requirements
   - OAuth scope definitions
   - Event Subscriptions configuration

4. **Matrix Appservice Protocol**:
   - Understanding as_token vs hs_token
   - Appservice registration format
   - Namespace claiming (user patterns)

* Raw Notes

## Timeline of Key Events

1. **Session start**: Continued from previous session, bridge infrastructure deployed but bot not responding
2. **Discovered IPv6 issue**: nginx errors showed connection refused to [::1]:8008
3. **Fixed nginx config**: Changed proxy_pass to 127.0.0.1
4. **Discovered bridge IPv6 issue**: Same problem in bridge homeserverUrl
5. **Fixed bridge config**: Changed to 127.0.0.1
6. **Deployed changes**: Used rsync + VPS build pattern
7. **Bridge still failing**: as_token rejected errors
8. **Tried debug logging**: Changed log level to debug
9. **Tried manual token testing**: curl commands to validate as_token
10. **Multiple re-registration attempts**: Unregister/register cycle in admin room
11. **Breakthrough decision**: Suggested wiping database and starting fresh
12. **User agreed**: "Ok, We're doing this on an old db version or something?"
13. **Wiped databases**: Both Matrix (RocksDB) and bridge (PostgreSQL)
14. **Fresh registration**: Registered appservice on clean database
15. **Bridge started successfully**: No more as_token errors!
16. **Bot responded to help command**: First successful interaction
17. **Configured Slack app**: User had to get boss approval for scopes
18. **Authenticated with Slack**: Sent xapp- and xoxb- tokens to bot
19. **Messages flowing Slack → Matrix**: Initial sync created ~50 rooms
20. **Tested bidirectional**: Sent message from Matrix, appeared in Slack
21. **Success**: Both directions working, feature deployed

## Error Messages Encountered

```
FTL The as_token was not accepted. Is the registration file installed in your homeserver correctly?
```
This was the persistent error that ultimately required database wipe to resolve.

```
connect() failed (111: Connection refused) while connecting to upstream, upstream: "http://[::1]:8008/..."
```
nginx trying to connect to IPv6 localhost when Matrix only listening on IPv4.

```
dial tcp [::1]:8008: connect: connection refused
```
Bridge trying to connect to IPv6 localhost when Matrix only listening on IPv4.

```
M_UNKNOWN_TOKEN: Unknown access token
```
When manually testing as_token via curl to Matrix API.

```
Expected code block in command body. Add --help for details.
```
User's first attempt at !admin appservices register command formatting.

## Quotes from Session

User: "Ok, We can't enable socket mode quite yet, what else should we do."
- Session started with blocked state on Slack app configuration

User: "can we just.. wipe out everything and start fresh?"
- Key moment where we decided to abandon debugging and go for fresh database

User: "Ok, We're doing this on an old db version or something? Like, we migrated and deployed.. we don't need anything on the current db"
- User's insight that led to the solution - fresh database eliminated stale state

User: "everytime we do this I have to email the boss, I'm going to get fired from the slack/pinball startup."
- Regarding Slack app scope changes requiring workspace admin approval

User: "Great, We are at that point. I'm on the slackbot webpage."
- Configuring Slack app for Socket Mode

User: "Ok, It looks like it's going with the old token? I can now see messages _from_ slack in matrix"
- First success! Slack → Matrix working

User: "All good, Test to and from #vlads-pad worked."
- Final confirmation of bidirectional success

## Bridge Startup Logs (Successful)

```
2025-10-26T19:19:54.580Z INF Initializing bridge built_at=0001-01-01T00:00:00Z go_version=go1.25.2 name=mautrix-slack version=v25.10
2025-10-26T19:19:54.591Z INF Starting bridge
2025-10-26T19:19:54.594Z INF Database is up to date current_version=9 db_section=matrix_state latest_known_version=9 oldest_compatible_version=3
2025-10-26T19:19:54.595Z INF Starting HTTP listener address=127.0.0.1:29319
2025-10-26T19:19:54.753Z INF No user logins found
2025-10-26T19:19:54.753Z INF Bridge started
```

Notice: No "as_token was not accepted" error! This is what success looks like.

## Initial Sync Output

Bridge automatically discovered and created Matrix rooms for ~50 Slack channels:
- Public channels (C0...)
- Private channels/groups (also C0...)
- Direct messages (D0...)
- Multi-party DMs

Each room creation logged as:
```
2025-10-26T19:27:19.843Z INF Matrix room created action="create matrix room" portal_id=TSW76H2Q0-CU7KVAT50 room_id=!ptrEGpKQlvpJtiFzq3:clarun.xyz
```

This automatic discovery and room creation is one of the most powerful features of the mautrix bridge framework.

## Configuration File Locations

On VPS (45.77.205.49):
- Matrix config: `/var/lib/matrix-continuwuity/continuwuity.toml`
- Matrix database: `/var/lib/matrix-continuwuity/db/` (RocksDB)
- Bridge config: `/var/lib/mautrix_slack/config/config.yaml`
- Bridge registration: `/var/lib/matrix-appservices/mautrix_slack_registration.yaml`
- Bridge database: PostgreSQL database `mautrix_slack` owned by user `mautrix_slack`
- Secrets: `/run/secrets/matrix-registration-token` (sops-decrypted)

## Nix Store Paths

- conduwuit binary: `/nix/store/8s4mvvw92rw7b7bkx13dzg3rxpi37bjv-matrix-continuwuity-0.5.0-rc.8/bin/conduwuit`
- mautrix-slack binary: `/nix/store/gw3i53cilaiqach3lb1n43vvp5wqicwz-mautrix-slack-25.10/bin/mautrix-slack`
- nginx: `/nix/store/zfkzs34vp39q8ikv582mkj5vj6jm0bpr-nginx-1.26.2/bin/nginx`

## Service Status

All services running and healthy:
```
● matrix-continuwuity.service - active (running)
● mautrix-slack.service - active (running)
● nginx.service - active (running)
● postgresql.service - active (running)
```

## Network Verification

```bash
# Matrix homeserver listening on IPv4 only
$ ss -tlnp | grep 8008
LISTEN 0      4096       127.0.0.1:8008       0.0.0.0:*

# Bridge listening on IPv4 only
$ ss -tlnp | grep 29319
LISTEN 0      4096       127.0.0.1:29319      0.0.0.0:*

# Nginx listening on public IPv4 + IPv6
$ ss -tlnp | grep :443
LISTEN 0      511        0.0.0.0:443          0.0.0.0:*
LISTEN 0      511           [::]:443             [::]:*
```

This shows the correct network binding: internal services on IPv4 loopback, nginx on all interfaces for HTTPS.

* Session Metrics
- Commits made: 0 (changes not yet committed)
- Files touched: 19 (though many are spec-kit framework updates unrelated to this feature)
- Core feature files modified: 2 (modules/dev-services.nix, CLAUDE.md)
- Lines changed in dev-services.nix: 12 lines (IPv4 addresses + debug log level)
- Deployment attempts: 6-7 (including failed attempts)
- Time to resolution: ~3 hours of active troubleshooting
- Database wipes: 2 (Matrix RocksDB, PostgreSQL)
- Slack app approval requests: 2 ("boss will fire me")
- Matrix rooms created: ~50 (automatic sync from Slack)
- Bridge restarts: 30+ (crash loop during debugging, then successful)
- Tests passing: 2/2 (bidirectional message flow verified in #vlads-pad channel)