#+TITLE: HQ Orchestrator Architecture, Orch Consensus Validation, and Beads Cleanup
#+DATE: 2026-01-11
#+KEYWORDS: hq, orchestrator, orch-consensus, beads, worker-cli, multi-agent, architecture, lego-brick
#+COMMITS: 4
#+COMPRESSION_STATUS: uncompressed

* Session Summary
** Date: 2026-01-11 (Continuation of multi-agent architecture work)
** Focus Area: HQ orchestrator design, architecture validation via orch consensus, and beads issue management

* Accomplishments
- [X] Named the orchestrator "HQ" (headquarters) - 2 chars, fits naming convention (bd, orch)
- [X] Defined HQ as a skill, not a binary - any agent with the skill can orchestrate
- [X] Ran orch consensus on bd comments vs JSONL for inter-agent messaging - unanimous support for bd comments
- [X] Ran orch consensus on overall architecture - validated "Lego brick" approach with gap analysis
- [X] Created HQ feature issue (skills-2l4e) with 4 sub-tasks
- [X] Filed 5 new issues from consensus gap analysis (garbage collection, context pruning, event notification, retry limits, state machine docs)
- [X] Closed skills-ms5 (message passing layer) - solved by bd comments decision
- [X] Closed skills-iusu (bd comments evaluation) - complete, unanimous consensus
- [X] Closed skills-s6y (Lego architecture epic) - MVP complete, architecture validated
- [X] Unblocked 17 design/research tasks (blocked count: 23 → 4)
- [X] Fixed worktree path bug - now uses absolute paths
- [X] Added scenario schema and test fixtures for benchmark harness
- [X] Completed session close protocol - all changes committed and pushed

* Key Decisions
** Decision 1: Name the orchestrator "HQ"
- Context: Needed a short, memorable name for the orchestration skill
- Options considered:
  1. helm - taken by Kubernetes package manager
  2. con (conductor) - too short, ambiguous
  3. deck - nautical theme but unclear
  4. cap (captain) - too generic
  5. ops - conflicts with operations
  6. hq (headquarters) - 2 chars, clear meaning
- Rationale: HQ is short (2 chars like bd, orch), universally understood, implies coordination/command center
- Impact: All orchestration work will use "hq" as the skill name and CLI prefix

** Decision 2: HQ is a skill, not a daemon/binary
- Context: How should the orchestrator be implemented?
- Options considered:
  1. Daemon/service watching for events
  2. CLI binary with polling
  3. Skill (markdown instructions + scripts) that any agent can load
- Rationale: A skill fits the "Lego brick" philosophy - no new infrastructure, any capable agent becomes an orchestrator by loading the skill
- Impact: Implementation focuses on SKILL.md design and helper scripts rather than new binaries

** Decision 3: Use bd comments for inter-agent messaging (unanimous consensus)
- Context: How should agents communicate status and coordinate?
- Options considered:
  1. Option A: Use existing bd issue comments (append-only, structured prefixes)
  2. Option B: Build new JSONL message transport layer
- Rationale: Orch consensus (flash-or, gemini, gpt, qwen) unanimously supported Option A:
  - Unified source of truth with issue tracker
  - Zero infrastructure overhead
  - Human-readable and debuggable
  - Context size manageable with --last N and periodic summarization
- Impact: Closed skills-ms5, no new message layer needed, focus on bd comment conventions

** Decision 4: Close skills-s6y (Lego epic) to unblock dependent work
- Context: 17 design/research issues blocked by the main architecture epic
- Rationale:
  - Core MVP complete: worker CLI, state machine, review-gate, branch isolation
  - Architecture validated by orch consensus
  - Remaining blocked items are design tasks that can proceed
- Impact: Blocked count dropped from 23 to 4, ready-to-work increased from 21 to 38

* Problems & Solutions
| Problem | Solution | Learning |
|---------|----------|----------|
| Relative worktree path failed from inside worktree | Changed worktreePath() to use findMainRepoDir() for absolute paths | Always use absolute paths for operations that may run from different working directories |
| Test suite didn't catch path bug because it bypassed done command | Bug discovered during manual spike testing | Manual integration tests catch what unit tests miss |
| 23 issues blocked by architecture epic | Closed epic after MVP complete, architecture validated | Epics should be closed when core work is done to unblock dependent tasks |
| Qwen returned empty response in orch consensus | Other 3 models provided clear responses | Multi-model consensus is robust to individual model failures |

* Technical Details

** Code Changes
- Total files modified: 15 (across all commits)
- Key files changed:
  - =src/worker/utils.nim= - Fixed worktreePath() to return absolute path using findMainRepoDir()
  - =src/worker/tests/test-worker.sh= - Added sqlite3 availability check
- New files created:
  - =docs/specs/scenario-schema.md= - YAML schema for agent capability test scenarios
  - =tests/scenarios/easy/add-factorial.yaml= - Easy difficulty test scenario
  - =tests/scenarios/medium/add-caching-to-api.yaml= - Medium difficulty scenario
  - =tests/scenarios/hard/fix-race-condition.yaml= - Hard difficulty scenario
  - =tests/fixtures/python-math-lib/= - Python fixture for testing
  - =tests/fixtures/flask-user-api/= - Flask API fixture placeholder

** Commands Used
#+begin_src bash
# Orch consensus queries
orch query --models flash-or,gemini,gpt,qwen "Support or challenge..."

# Beads management
bd close skills-ms5 --reason="Solved by bd comments approach..."
bd close skills-s6y --reason="MVP complete: worker CLI, state machine..."
bd blocked  # Check remaining blocked issues
bd stats    # Verify impact of closures

# Session close protocol
bd sync
git add <files>
git commit -m "..."
git push
#+end_src

** Architecture Notes
- HQ skill structure: SKILL.md (instructions) + helper scripts (hq-status, hq-spawn, hq-check)
- Worker template: System prompt for workers operating in worktree context
- BD comments as message layer: Structured prefixes (status:, plan:, agent:), --last N filtering, periodic summarization
- Gaps identified by consensus: garbage collection, locking/race conditions, context pruning, cost budgets, retry limits, state machine invariants

* Process and Workflow

** What Worked Well
- Orch consensus for validating architecture - multiple perspectives caught gaps
- Manual spike testing - found real bug that tests missed
- Batch closing related issues - efficient beads management
- Session compaction preserved essential context for continuation

** What Was Challenging
- Balancing design depth vs implementation progress
- Determining when an epic is "done enough" to close
- Qwen returning empty response (compensated by other models)

* Learning and Insights

** Technical Insights
- Absolute paths essential for operations that run from different directories
- Git worktree path resolution is context-dependent
- Multi-model consensus provides robust architectural validation

** Process Insights
- Orch consensus is effective for binary decisions (Option A vs B)
- Closing blocking epics when core work is done unblocks significant downstream work
- Manual integration spikes catch issues automated tests miss

** Architectural Insights
- "Skill as orchestrator" pattern avoids infrastructure complexity
- Existing primitives (bd comments) often sufficient - resist building new layers
- Unix philosophy applies to AI orchestration: small, composable, text-based

* Context for Future Work

** Open Questions
- How should HQ handle concurrent human + agent modifications?
- What's the right heartbeat/timeout for stale worker detection?
- How to summarize bd comment history without losing critical context?
- Should HQ be model-agnostic or tuned for specific models?

** Next Steps
Ready-to-work issues (now unblocked):
- skills-21ka: Design HQ SKILL.md - orchestration instructions
- skills-cg7c: Design worker system prompt template
- skills-3j55: Create hq-status script
- skills-w9a4: Garbage collection / janitor for orphaned workers
- skills-8hyz: Context pruning for bd comments
- skills-vdup: Retry limits and escalation policy
- skills-s2bt: State machine invariants documentation

** Related Work
- Previous worklog: [[file:2026-01-11-worker-cli-cleanup-refactors.org][Worker CLI Cleanup]] (same day, earlier session)
- Previous worklog: [[file:2026-01-10-multi-agent-lego-architecture-design.org][Multi-Agent Lego Architecture Design]]
- Orch consensus output: /tmp/claude/-home-dan-proj-skills/tasks/b019872.output (architecture validation)
- Orch consensus output: /tmp/claude/-home-dan-proj-skills/tasks/b6d9640.output (bd comments decision)

* Raw Notes

** Orch Consensus Summary: BD Comments vs JSONL
All 4 models (flash-or, gemini, gpt, qwen) supported Option A (bd comments):
- "Fits your 'Lego bricks' principle" (GPT)
- "Unified source of truth means human developers and AI agents share the same context" (Gemini Flash)
- "Avoid the 'distributed systems' trap of syncing a separate message layer" (Gemini)
- "Lower maintenance overhead... reduces complexity and potential failure points" (Qwen)

Recommended mitigations for context size:
- --last N filtering
- Structured prefixes (status:, plan:, agent:)
- Periodic summarization by manager agent
- State checkpoints in pinned comments

** Orch Consensus Summary: Architecture Validation
Support with identified gaps:

1. Garbage collection - orphaned worktrees, stuck workers, abandoned locks
2. Locking/race conditions - file locks with TTL, idempotent operations
3. Context pruning - summarize closed issue history
4. Cost/budget controls - token limits, spawn limits
5. Retry limits - escalate to human after N failures
6. State machine invariants - document allowed transitions
7. Event notification - inotify/watcher vs polling tradeoff

** Beads Status After Session
- Open: 42 (was 44)
- Blocked: 4 (was 23)
- Ready to work: 38 (was 21)
- Closed: 200 (was 197)

Remaining blocked issues (legitimate dependencies):
- skills-2l4e (HQ) - blocked by its sub-tasks
- skills-kg7 (Desktop automation) - blocked by AT-SPI chain
- skills-hf1 (Cross-agent portability) - blocked by research tasks

* Session Metrics
- Commits made: 4 (excluding bd sync commits)
- Files touched: 15
- Lines added/removed: +660/-26
- Issues created: 6 (HQ feature + 5 gap tasks)
- Issues closed: 3 (ms5, iusu, s6y)
- Issues unblocked: 17