ops-jrz1/specs/003-maubot-integration/spec.md

Feature Specification: Matrix Bot Framework (Maubot) Integration

Feature Branch: 003-maubot-integration Created: 2025-10-26 Status: Draft Input: User description: "Begin maubot feature spec. instagram bot is one of our goals."

Clarifications

Session 2025-10-26

• Q: Instagram bot activation behavior - should it respond to all Instagram URLs, only when mentioned, or in designated rooms? → A: Bot responds to Instagram URLs only in designated bot-enabled rooms
• Q: Bot error notification method - how should errors be communicated to administrators? → A: Error notification behavior based on severity levels (DEBUG/INFO logs only, WARN logs + dashboard visibility, ERROR/CRITICAL logs + dashboard + Matrix admin room notifications)
• Q: Room enablement mechanism - how do administrators enable bot in specific rooms? → A: Edit bot configuration file with room IDs, restart bot instance
• Q: Admin notification room configuration - should each bot have dedicated admin room, shared room, or reuse homeserver admin room? → A: Reuse Matrix homeserver admin room for bot ERROR/CRITICAL notifications
• Q: Management interface authentication - single shared account, multi-user, or Matrix homeserver auth? → A: Single shared admin account (username/password configured in sops-nix secrets)

User Scenarios & Testing (mandatory)

User Story 1 - Instagram Content Sharing to Matrix (Priority: P1)

A team member shares an Instagram post URL in a Matrix room, and the bot automatically fetches and displays the content (image, caption, metadata) directly in the chat, allowing team members to view and discuss Instagram content without leaving Matrix.

Why this priority: This is the core value proposition - bringing Instagram content into team communication. Demonstrates immediate utility of the bot framework and validates the integration works correctly.

Independent Test: Can be fully tested by posting an Instagram URL in a Matrix room and verifying the bot responds with content preview, delivering immediate value as an Instagram content viewer.

Acceptance Scenarios:

1. Given Instagram bot is enabled in a specific Matrix room, When user posts "https://instagram.com/p/ABC123/" in that room, Then bot responds within 5 seconds with image, caption, and post metadata (likes, comments count)
2. Given Instagram bot is NOT enabled in a Matrix room, When user posts Instagram URL in that room, Then bot ignores the URL and does not respond
3. Given bot receives Instagram URL in enabled room, When content is a video, Then bot provides video thumbnail, caption, and download link
4. Given bot receives Instagram URL in enabled room, When content is a carousel (multiple images), Then bot displays all images in sequence with navigation
5. Given bot receives Instagram profile URL in enabled room, When URL is "https://instagram.com/username", Then bot displays profile info (bio, follower count, recent posts preview)
6. Given bot encounters rate limiting in enabled room, When too many requests in short period, Then bot queues request and notifies user of delay

---

User Story 2 - Bot Management Interface (Priority: P2)

Platform administrators can configure, start, stop, and monitor bots through a web-based management interface without editing configuration files or restarting services.

Why this priority: Essential for operational management and enables non-developer administrators to manage bots. Required for long-term maintainability but bot can work without it initially.

Independent Test: Can be tested by accessing management interface, creating a test bot instance, and verifying it appears in Matrix - demonstrates full bot lifecycle management.

Acceptance Scenarios:

1. Given administrator accesses Maubot management UI, When they log in with shared admin credentials, Then dashboard displays all bot instances, their status, and health metrics
2. Given administrator wants to deploy Instagram bot, When they upload maubot plugin file (.mbp), Then plugin appears in available plugins list
3. Given plugin is uploaded, When administrator creates new bot instance with Matrix user credentials and room subscription list, Then bot appears online in Matrix within 30 seconds and only responds in configured rooms
4. Given administrator wants to change enabled rooms, When they edit bot configuration file with new room IDs and restart bot instance, Then bot begins responding only in newly configured rooms
5. Given bot is running, When administrator clicks "Stop" button, Then bot goes offline and stops responding to commands
6. Given bot encounters error, When viewing bot logs in UI, Then error messages are displayed with timestamps, severity level, and context
7. Given bot experiences CRITICAL error, When error occurs, Then notification is sent to Matrix homeserver admin room with error details and affected bot instance

---

User Story 3 - Bot Framework Service Reliability (Priority: P2)

The Maubot service starts automatically on server boot, maintains bot instances across restarts, and recovers from failures without manual intervention.

Why this priority: Critical for production use but can be validated after basic functionality works. Prevents the bot framework from being a maintenance burden.

Independent Test: Can be tested by rebooting the server and verifying Maubot service auto-starts and all bot instances resume operation automatically.

Acceptance Scenarios:

1. Given server reboots, When system comes back online, Then Maubot service starts automatically within 2 minutes and all bot instances reconnect to Matrix
2. Given Matrix homeserver restarts, When homeserver is available again, Then bot instances re-establish connections and resume operation without manual intervention
3. Given bot instance crashes, When Maubot detects failure, Then service attempts automatic restart with exponential backoff
4. Given bot encounters persistent error (ERROR/CRITICAL severity), When restart attempts fail, Then service logs detailed diagnostics, updates dashboard status, and sends notification to Matrix homeserver admin room
5. Given database connection lost, When connectivity is restored, Then Maubot reconnects automatically and restores bot state

---

User Story 4 - Additional Bot Deployment (Priority: P3)

Platform administrators can deploy additional custom bots beyond Instagram bot by uploading plugin files and configuring bot instances, enabling extensible bot functionality for future team needs.

Why this priority: Demonstrates platform extensibility and future-proofs the investment, but not required for initial value delivery. Can be added after Instagram bot proves value.

Independent Test: Can be tested by deploying a simple echo bot or reaction bot from maubot plugin repository and verifying it works independently.

Acceptance Scenarios:

1. Given administrator has custom maubot plugin (.mbp file), When they upload via management interface, Then plugin is validated and added to available plugins
2. Given plugin requires configuration, When creating bot instance, Then administrator can provide plugin-specific settings through UI
3. Given multiple bot instances exist, When administrator views dashboard, Then all bots are clearly listed with their types, status, and resource usage
4. Given bot requires database storage, When bot instance is created, Then Maubot automatically provisions isolated database for that bot
5. Given plugin has dependencies, When uploading plugin, Then Maubot validates dependencies and reports missing requirements

---

Requirements (mandatory)

Functional Requirements

• FR-001: System MUST extract and deploy maubot module from ops-base repository to ops-jrz1 infrastructure
• FR-002: System MUST integrate Maubot with existing conduwuit Matrix homeserver on clarun.xyz
• FR-003: System MUST provide web-based management interface on dedicated port (default: 29316) accessible to platform administrators via single shared admin account credentials stored in sops-nix secrets
• FR-004: Maubot service MUST support automatic startup on system boot and auto-recovery from failures
• FR-005: System MUST support Instagram bot plugin deployment with content fetching capabilities
• FR-006: Instagram bot MUST fetch and display images, videos, captions, and metadata from Instagram URLs posted only in designated bot-enabled Matrix rooms (bot ignores URLs in rooms where it is not explicitly enabled)
• FR-007: Instagram bot MUST handle rate limiting gracefully with user-friendly error messages
• FR-008: System MUST support multiple bot instances running concurrently with isolated configurations (architecture supports 3+ instances per SC-002, production deploys 1 instance initially per quickstart.md)
• FR-009: System MUST persist bot configurations and state to survive service restarts
• FR-010: Administrators MUST be able to configure bot room subscriptions by editing bot configuration file with Matrix room IDs and restarting the bot instance
• FR-011: System MUST provide health monitoring for bot instances with status indicators (health check API endpoint and dashboard status display via management interface)
• FR-012: System MUST integrate with existing sops-nix secrets management for bot credentials
• FR-013: System MUST support uploading and deploying additional maubot plugins (.mbp files) - functionality inherited from ops-base maubot.nix module, validated in T029
• FR-014: System MUST provide logging capabilities for bot activity and errors accessible via management interface with severity-based propagation (DEBUG/INFO to logs only, WARN to logs and dashboard, ERROR/CRITICAL to logs, dashboard, and Matrix homeserver admin room)
• FR-015: Bot instances MUST authenticate with Matrix homeserver using registration tokens (conduwuit compatibility requirement, shared secret not supported)
• FR-016: System MUST support per-bot database storage with automatic provisioning

Key Entities

• Maubot Service: Plugin-based Matrix bot framework that manages multiple bot instances, provides management interface, and handles Matrix homeserver integration
• Bot Instance: Individual bot deployment with specific configuration, Matrix user account, and plugin assignment (e.g., "instagram-bot-1")
• Plugin: Packaged bot functionality (.mbp file) containing code, metadata, and dependencies (e.g., Instagram content fetcher, echo bot, reaction bot)
• Bot Configuration: Settings specific to bot instance including Matrix credentials, plugin settings, room subscriptions (list of enabled room IDs), and command prefixes
• Management Interface: Web UI for administrators to create, configure, monitor, and control bot instances, displaying logs with severity levels and real-time status updates
• Admin Notification: ERROR and CRITICAL level bot notifications sent to existing Matrix homeserver admin room (shared with other platform notifications)
• Bot Database: Per-instance isolated SQLite database for plugin state and data persistence

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: Instagram bot responds to Instagram URLs with content preview within 5 seconds under normal conditions
• SC-002: System supports at least 3 concurrent bot instances without performance degradation
• SC-003: Maubot service maintains 99% uptime over 7-day testing period
• SC-004: Bot instances automatically recover within 2 minutes after service restart
• SC-005: Administrators can deploy a new bot instance from scratch in under 10 minutes
• SC-006: Instagram bot successfully fetches content for 95% of public Instagram post URLs
• SC-007: Management interface loads and displays bot status within 2 seconds
• SC-008: System handles server reboot without data loss or manual intervention required

Validation Note: SC-001, SC-002, SC-003, SC-004, SC-008 have explicit task validation (T026, T042, T038, T034, T034). SC-005, SC-006, SC-007 are measured during the 7-day operational validation period (T038) and documented in deployment worklog (T044).

Scope (mandatory)

In Scope

• Extract and adapt maubot.nix module from ops-base to ops-jrz1
• Configure Maubot to integrate with conduwuit Matrix homeserver
• Deploy Instagram bot plugin as primary use case
• Set up management web interface with authentication
• Implement health monitoring and auto-recovery mechanisms
• Configure sops-nix secrets for bot credentials
• Document bot deployment and management procedures including room subscription configuration workflow
• Support for uploading additional maubot plugins

Out of Scope

• Custom Instagram bot development (use existing maubot Instagram plugin from community)
• Migration of other bots from ops-base besides Instagram bot
• Advanced analytics or metrics dashboard for bot performance
• Multi-homeserver support (only clarun.xyz)
• Custom plugin development beyond Instagram bot deployment
• Mobile app for bot management (web interface only)
• Automatic Instagram authentication (manual token provisioning acceptable)
• Real-time Instagram feed monitoring or notifications

Constraints (mandatory)

Technical Constraints

• Must work with conduwuit Matrix homeserver (ops-base used continuwuity, may require compatibility testing)
• Limited to Python 3.11 for maubot runtime (nixpkgs availability)
• Instagram bot functionality depends on Instagram API/scraping availability and rate limits
• Must adapt from ops-base VM-based deployment pattern to ops-jrz1 VPS single-host pattern
• Dependent on deprecated olm-3.2.16 library for Matrix encryption (known CVEs, acceptable risk documented in ops-base)

Operational Constraints

• Deployment must not disrupt existing services (Matrix homeserver, Slack bridge, Forgejo)
• Management interface must be secured (single admin account authentication, localhost-only access)
• Management interface credentials must be stored in sops-nix encrypted secrets
• Bot Matrix accounts require registration tokens from homeserver
• Instagram tokens may require periodic renewal based on Instagram API policies

Resource Constraints

• Maubot service limited to 512M memory (as per ops-base configuration)
• Additional database space required for bot state (estimated <100MB initially)
• Management interface port 29316 must not conflict with existing services

Dependencies (mandatory)

External Dependencies

• ops-base repository access to extract maubot.nix module and documentation
• Instagram bot plugin from maubot community or ops-base implementation
• Instagram authentication tokens (if required by current Instagram API policies)
• Matrix homeserver registration token for bot user creation

Internal Dependencies

• conduwuit Matrix homeserver must be operational on clarun.xyz
• sops-nix secrets management must be configured for bot credentials
• SQLite for bot state storage (decision per plan.md research: lightweight isolation better than shared PostgreSQL)
• Existing NixOS infrastructure and deployment patterns

Blocking Issues

• Need to verify conduwuit compatibility with maubot (ops-base used continuwuity)
• Need to assess current Instagram API access requirements and scraping feasibility
• Need to extract and adapt ops-base module configuration options from services.matrix-vm.maubot to services.dev-platform.maubot

Assumptions (mandatory)

• Instagram content fetching remains technically feasible (no major Instagram API changes blocking access)
• Maubot works with conduwuit Matrix homeserver with minimal or no modifications
• ops-base maubot module can be adapted to VPS deployment with reasonable effort
• Instagram bot plugin from ops-base is functional and can be reused or community plugin exists
• Team accepts olm-3.2.16 security risk with documented mitigation plan (migration to vodozemac when available)
• Bot traffic will remain under Instagram rate limits for small team usage (<100 requests/hour)
• Single VPS deployment sufficient (no distributed bot architecture needed)
• Single shared admin account sufficient for initial deployment (no multi-user management required)

Non-Goals (optional)

• Automated Instagram post monitoring or scheduled fetching
• Direct posting to Instagram from Matrix (read-only integration)
• Instagram DM integration or two-way messaging
• Advanced content moderation or filtering
• Custom Instagram analytics or engagement tracking
• Multi-tenant bot hosting for external teams
• Commercial Instagram API integration (acceptable to use community scraping approaches)
• Real-time Instagram notifications or webhooks

Known Limitations (optional)

The following edge cases are known limitations not addressed in MVP scope:

• Deleted/private Instagram posts: Bot does not handle posts that become private or deleted after initial fetch (content remains in Matrix chat history)
• Instagram rate limiting: System may experience delays during high-traffic periods (429 responses). FR-007 requires graceful handling with user notifications.
• Matrix account credential expiry: Bot user account credentials are managed via registration tokens and do not expire automatically. Manual re-authentication required if revoked.
• Instagram story URLs: 24-hour expiry stories not supported (yt-dlp limitation for ephemeral content)
• Command collision: Multiple bot instances in same room may respond to overlapping triggers. Recommendation: enable only one bot per room or use distinct command prefixes.
• Age-restricted/geo-blocked content: Instagram content with access restrictions may fail to fetch depending on VPS location and yt-dlp capabilities
• Management interface connection loss: If Maubot loses connection to Matrix homeserver, bot instances stop responding until connection restored (monitored via health checks in FR-011)
• Database corruption: No automated backup/recovery. Recommendation: implement manual backup procedure for /var/lib/maubot/ during operational period.

Risks (optional)

Technical Risks

• Risk: Instagram API/scraping methods may break with Instagram updates

  • Mitigation: Document bot as best-effort, plan for periodic maintenance, monitor Instagram bot community for updates

• Risk: Conduwuit compatibility issues with maubot not discovered until integration

  • Mitigation: Test maubot registration and basic functionality early in implementation phase

• Risk: olm-3.2.16 vulnerabilities may be exploited

  • Mitigation: Follow ops-base mitigation strategy - monitor for vodozemac migration, limit bot network exposure, document accepted risk

Operational Risks

• Risk: Instagram rate limiting may impact bot responsiveness during high usage

  • Mitigation: Implement request queuing, user notifications for delays, consider rate limit monitoring

• Risk: Bot management interface security breach could compromise Matrix homeserver

  • Mitigation: Require strong authentication, limit network exposure, regular security audits, use sops-nix for credential storage

• Risk: Bot instance failure may go unnoticed without monitoring

  • Mitigation: Implement health checks, automated restarts, log monitoring, administrator alerts for persistent failures

Clarified Requirements (resolved 2025-10-26)

Instagram Authentication Approach

Decision: Use community scraping methods (instaloader, yt-dlp) for Instagram content fetching.

Rationale: Easier to set up immediately without requiring Facebook developer account approval. Acceptable for internal team use with understanding that scraping methods may require periodic updates if Instagram changes their interface.

Management Interface Network Exposure

Decision: Restrict management interface to localhost only, requiring SSH tunnel for remote administration.

Rationale: Maximizes security by eliminating network attack surface. Administrators already have SSH access for deployment, so tunnel setup is acceptable operational overhead for the security benefit.

Bot Instance Quantity Planning

Decision: Support single Instagram bot instance initially (1 instance).

Rationale: Minimal resource requirements, proves concept quickly, demonstrates value before scaling. Architecture can support additional instances later if needed without major rework.

Note: SC-002 requires validating 3-instance capability during testing to ensure architecture can scale when needed, but production deployment starts with single instance.