211 lines
9.1 KiB
Org Mode
211 lines
9.1 KiB
Org Mode
#+TITLE: Multi-Lens Code Review Workflow Testing on Orch
|
|
#+DATE: 2025-12-26
|
|
#+KEYWORDS: code-review, lenses, orch, beads, molecules, LLM-in-the-loop, parallel-agents
|
|
#+COMMITS: 3
|
|
#+COMPRESSION_STATUS: uncompressed
|
|
|
|
* Session Summary
|
|
** Date: 2025-12-26 (Continuation of 2025-12-24 session)
|
|
** Focus Area: Testing multi-lens code review workflow on orch codebase
|
|
|
|
* Accomplishments
|
|
- [X] Recovered session context from compaction (continued from 2025-12-24 ADR/lenses session)
|
|
- [X] Ran 3 code review lenses (smells, dead-code, redundancy) on orch/src/orch/ via parallel Task agents
|
|
- [X] Synthesized findings across all lenses using LLM judgment
|
|
- [X] Filed 4 actionable beads issues in orch repo based on synthesis
|
|
- [X] Validated the LLM-in-the-loop pattern for code review issue filing
|
|
- [X] Synced beads in both skills and orch repos
|
|
- [ ] Push to remote blocked (git server 192.168.1.108:3000 unavailable)
|
|
|
|
* Key Decisions
|
|
** Decision 1: Use parallel Task agents for lens execution
|
|
- Context: Need to run multiple lenses efficiently without sequential bottleneck
|
|
- Options considered:
|
|
1. Sequential lens execution - simpler but slower
|
|
2. Parallel Task agents - faster, demonstrates multi-agent pattern
|
|
3. Single agent running all lenses - loses multi-model perspective
|
|
- Rationale: Parallel agents match the "50-100 agents at once" vision from Yegge's patterns
|
|
- Impact: 3x faster execution, demonstrates scalable pattern for larger codebases
|
|
|
|
** Decision 2: Synthesize findings before filing issues
|
|
- Context: Raw lens output produces many overlapping findings
|
|
- Options considered:
|
|
1. Mechanical parsing - file every finding as separate issue
|
|
2. LLM synthesis - agent applies judgment, groups related, sets priorities
|
|
- Rationale: LLM-in-the-loop pattern per Yegge - agent stays involved at every step
|
|
- Impact: Reduced 20+ raw findings to 4 actionable, prioritized issues
|
|
|
|
** Decision 3: File issues in target repo, not skills repo
|
|
- Context: Lenses run from skills repo, findings belong to target codebase
|
|
- Rationale: Issues should live where the work will be done
|
|
- Impact: orch repo now has actionable refactoring tasks with proper context
|
|
|
|
* Problems & Solutions
|
|
| Problem | Solution | Learning |
|
|
|---------|----------|----------|
|
|
| bd sync in orch failed with prefix mismatch | molecules.jsonl contains skills- prefixed proto tasks | bd-k2wg filed: mol catalog needs hierarchical loading |
|
|
| qwen returns empty content on some prompts | Use gpt/gemini as alternatives, filed orch-loq bug | Content filter issue or API quirk with certain prompt structures |
|
|
| Remote git server unavailable | Local commits preserved, push pending | Distributed git works - local work continues despite server outage |
|
|
| Context compaction lost previous session details | Summary preserved key decisions and file locations | Comprehensive summaries enable seamless session continuity |
|
|
|
|
* Technical Details
|
|
|
|
** Code Changes
|
|
- Total files modified: 16 (across session including prior context)
|
|
- Key commits this session:
|
|
- 1e64515 bd sync: 2025-12-26 01:57:15
|
|
- 9624873 update: code-review proto with LLM-in-the-loop pattern
|
|
- fb15000 refactor: restructure for cross-repo deployment
|
|
|
|
** Issues Filed in Orch
|
|
Four synthesized issues based on multi-lens analysis:
|
|
|
|
1. ~orch-buy~ [P2] refactor: create ModelSpec dataclass for provider parsing
|
|
- Addresses primitive obsession (smells) and duplication (redundancy)
|
|
- Same model:stance parsing done in 3+ places in cli.py
|
|
|
|
2. ~orch-bp3~ [P2] refactor: extract AttachmentProcessor from llm_adapter.py
|
|
- Addresses bloat (500+ line file) and complexity (deep nesting)
|
|
- llm_adapter.py at 590 lines, violates SRP
|
|
|
|
3. ~orch-esa~ [P2] refactor: split cli.py into command handlers
|
|
- Addresses bloat (600+ lines) and smells (flag arguments)
|
|
- CLI file handling multiple concerns
|
|
|
|
4. ~orch-loq~ [P2] bug: qwen returns empty content on some prompts
|
|
- Discovered during lens testing
|
|
- Possibly content filter or API issue
|
|
|
|
** Lens Output Highlights
|
|
|
|
*** Smells Lens Findings
|
|
- Deep nesting in stdin argument parsing (cli.py:314-327)
|
|
- Primitive obsession with model:stance string parsing
|
|
- Feature envy in llm_adapter.py (get_model_capabilities unused param)
|
|
- Flag arguments changing behavior (format string in show_session)
|
|
- Overloaded "status" variable name
|
|
|
|
*** Dead-Code Lens Findings
|
|
- Duplicate imports in llm_adapter.py (lines 10-12 vs 127-130)
|
|
- Empty TYPE_CHECKING block with only pass
|
|
- Unused config parameter in get_model_capabilities()
|
|
- Re-exports for "backward compatibility" with no external consumers
|
|
- Suspicious logic in use_llm_backend() comment vs implementation
|
|
|
|
*** Redundancy Lens Findings
|
|
- Temperature validation duplicated in config.py and validation.py
|
|
- Alias resolution chain: cli.py -> llm_adapter.py -> models_registry.py
|
|
- Error printing pattern repeated in cli.py (2x)
|
|
- Cancellation handling duplicated in chat() and consensus()
|
|
- Magic number 300.0 timeout in 3 places
|
|
- Response content filtering pattern repeated 4 times
|
|
|
|
** Commands Used
|
|
#+begin_src bash
|
|
# Run parallel lens agents via Task tool
|
|
# Each agent independently explored orch/src/orch/ and analyzed findings
|
|
|
|
# Sync beads
|
|
bd sync
|
|
|
|
# Check orch issues
|
|
cd /home/dan/proj/orch && bd list --status=open
|
|
|
|
# Attempt push (failed - server down)
|
|
git push
|
|
#+end_src
|
|
|
|
** Architecture Notes
|
|
- Parallel Task agents with subagent_type=Explore work well for lens analysis
|
|
- LLM synthesis step is crucial - raw findings too noisy for direct issue filing
|
|
- Proto workflow (skills-fvc) provides structure but agent applies judgment
|
|
- molecules.jsonl cross-repo loading has gaps (bd-k2wg filed)
|
|
|
|
* Process and Workflow
|
|
|
|
** What Worked Well
|
|
- Parallel agent execution - 3 lenses ran simultaneously
|
|
- Context recovery from compaction - summary preserved key decisions
|
|
- LLM-in-the-loop synthesis - reduced noise, grouped related findings
|
|
- Filing issues in target repo - proper ownership of work items
|
|
- Beads tracking - issues have context and history
|
|
|
|
** What Was Challenging
|
|
- Remote server outage prevented push (12 local commits pending)
|
|
- bd sync prefix mismatch when molecules.jsonl loaded in wrong repo
|
|
- Session compaction lost some details but summary covered essentials
|
|
- qwen model returning empty responses on some prompts
|
|
|
|
* Learning and Insights
|
|
|
|
** Technical Insights
|
|
- Multi-lens analysis reveals overlapping concerns (bloat + smells often co-occur)
|
|
- Parallel agents can explore same codebase without conflicts
|
|
- LLM judgment at filing step prevents issue spam
|
|
- Model quirks (qwen empty responses) need fallback strategies
|
|
|
|
** Process Insights
|
|
- "30-40% on code health" pattern from Yegge validated
|
|
- Multi-round convergent review could continue refining
|
|
- Proto templates provide structure without rigidity
|
|
- Session summaries enable context recovery after compaction
|
|
|
|
** Architectural Insights
|
|
- molecules.jsonl deployment to ~/.beads/ works but catalog loading incomplete
|
|
- Cross-repo workflow: lenses from skills, issues in target repo
|
|
- Beads prefix system creates friction when templates cross repos
|
|
- Home-manager deployment enables system-wide tool availability
|
|
|
|
* Context for Future Work
|
|
|
|
** Open Questions
|
|
- How to handle molecules.jsonl prefix conflicts? (bd-k2wg)
|
|
- Should lenses output structured JSON for easier parsing?
|
|
- How many rounds of lens passes before diminishing returns?
|
|
- Best strategy for qwen empty response fallback?
|
|
|
|
** Next Steps
|
|
- Implement bd mol catalog hierarchical loading (bd-k2wg)
|
|
- Run bloat lens on orch (skipped this session due to context focus)
|
|
- Test lenses on larger codebase (beads itself?)
|
|
- Add --synthesize flag to orch for automatic multi-model synthesis
|
|
|
|
** Related Work
|
|
- [[file:2025-12-24-adr-revision-lsp-research-code-audit.org][2025-12-24 ADR Revisions, LSP Research, Code Audit]] - Created lenses and proto
|
|
- Yegge "Vibe Coding" patterns - 30-40% code health, multi-round convergent review
|
|
- beads molecules feature - bd mol, bd wisp, bd pour commands
|
|
- orch multi-model consensus - Used for lens execution
|
|
|
|
* Raw Notes
|
|
- Session recovered from context compaction mid-conversation
|
|
- Previous session created lenses/bloat.md, smells.md, dead-code.md, redundancy.md
|
|
- Proto skills-fvc with 7 child tasks for structured code review workflow
|
|
- Deployed to ~/.config/lenses/ and ~/.beads/molecules.jsonl via home-manager
|
|
- Steve Yegge patterns influencing design:
|
|
- Software is throwaway (<1 year shelf life)
|
|
- 50-100 agents at once prediction
|
|
- 30-40% time on code health passes
|
|
- Multi-round convergent review (4-5 passes)
|
|
- "Land the Plane" protocol
|
|
|
|
** Lens Analysis Pattern
|
|
1. Agent explores target codebase structure
|
|
2. Reads key files (focus on largest/most complex)
|
|
3. Applies lens-specific criteria
|
|
4. Reports findings with severity and location
|
|
5. Parent agent synthesizes across all lenses
|
|
6. Parent applies judgment to file grouped, prioritized issues
|
|
|
|
** Issues with Current Flow
|
|
- molecules.jsonl loaded globally shows proto tasks in all projects
|
|
- Need bd mol catalog to filter by template label
|
|
- Prefix collision when syncing across repos
|
|
|
|
* Session Metrics
|
|
- Commits made: 3
|
|
- Files touched: 16
|
|
- Lines added/removed: +1801/-26
|
|
- Tests added: 0
|
|
- Issues filed: 4 (in orch repo)
|
|
- Agents spawned: 3 (parallel lens analysis)
|