docs: worklog for wayland desktop automation session
This commit is contained in:
dan
parent
906f2bc7ee
commit
e366343dd7
|
|
@ -0,0 +1,259 @@
|
|||
#+TITLE: Wayland Desktop Automation: The Seeing Problem
|
||||
#+DATE: 2025-12-17
|
||||
#+KEYWORDS: wayland, niri, desktop-automation, at-spi, vision-model, accessibility, ydotool, seeing-problem
|
||||
#+COMMITS: 4
|
||||
#+COMPRESSION_STATUS: uncompressed
|
||||
|
||||
* Session Summary
|
||||
** Date: 2025-12-17
|
||||
** Focus Area: Exploring desktop automation solutions for Wayland/niri compositor
|
||||
|
||||
* Accomplishments
|
||||
- [X] Fixed orch skill documentation - updated to use globally installed CLI instead of ~cd ~/proj/orch && uv run~
|
||||
- [X] Closed skills-d87 (orch skill documentation-only bug)
|
||||
- [X] Created epic skills-kg7 for "Desktop automation for Wayland/niri"
|
||||
- [X] Framed "the seeing problem" - how AI agents understand screen content
|
||||
- [X] Investigated AT-SPI accessibility framework on NixOS
|
||||
- [X] Discovered AT-SPI is disabled (~NO_AT_BRIDGE=1~, ~GTK_A11Y=none~)
|
||||
- [X] Created task skills-pdg documenting AT-SPI enablement requirements
|
||||
- [X] Created benchmark tasks for both vision and AT-SPI paths
|
||||
- [X] Conducted first vision model benchmark experiment with ChatGPT screenshot
|
||||
- [X] Identified "the acting problem" - discovered no mouse input tool installed
|
||||
- [X] Verified ydotool would work (user already in uinput group)
|
||||
- [ ] ydotool installation task not yet created
|
||||
|
||||
* Key Decisions
|
||||
** Decision 1: Hybrid approach for UI understanding
|
||||
- Context: Need to understand what's on screen (element types, coordinates) for automation
|
||||
- Options considered:
|
||||
1. AT-SPI only - structured data, precise coords, but requires system config + app compliance
|
||||
2. Vision model only - universal coverage, but coordinate precision unknown
|
||||
3. Hybrid - benchmark both, use appropriately
|
||||
- Rationale: Both paths have tradeoffs; measuring both allows informed decisions per use case
|
||||
- Impact: Created parallel benchmark tasks, AT-SPI becomes opt-in system config
|
||||
|
||||
** Decision 2: Boot-time option for AT-SPI
|
||||
- Context: AT-SPI adds runtime overhead to all GTK/Qt apps
|
||||
- Options considered:
|
||||
1. Always-on accessibility
|
||||
2. Boot-time toggle
|
||||
3. Session-level toggle
|
||||
- Rationale: Overhead only worth paying when automation is needed
|
||||
- Impact: Documented in skills-pdg as future implementation direction
|
||||
|
||||
** Decision 3: Frame as "the seeing problem"
|
||||
- Context: Needed clear terminology for the capability gap
|
||||
- Rationale: "Seeing" captures the essence - understanding what's visible on screen
|
||||
- Impact: Epic and tasks now organized around this framing; led to discovering complementary "acting problem"
|
||||
|
||||
* Problems & Solutions
|
||||
| Problem | Solution | Learning |
|
||||
|---------|----------|----------|
|
||||
| AT-SPI test crashed with "AT-SPI: Couldn't connect to accessibility bus" | AT-SPI needs to be running at session start; can't spin up ad-hoc | AT-SPI is session-level, not on-demand |
|
||||
| Started fresh AT-SPI bus showed 0 apps | Existing apps connected to dbus at their startup, don't see new bus | Apps must start AFTER AT-SPI is enabled |
|
||||
| bd sync failed with worktree error | Manually committed beads files with git add/commit | bd sync has edge cases; manual fallback works |
|
||||
| Couldn't test vision model coordinate predictions | No mouse click tool installed (ydotool missing) | "Seeing" is only half the problem; "acting" matters too |
|
||||
|
||||
* Technical Details
|
||||
|
||||
** Code Changes
|
||||
- Total files modified: 4
|
||||
- Key files changed:
|
||||
- ~skills/orch/SKILL.md~ - Removed ~cd ~/proj/orch && uv run~ prefix from all examples
|
||||
- ~skills/orch/README.md~ - Updated prerequisites and examples
|
||||
- Beads changes:
|
||||
- ~.beads/issues.jsonl~ - Added skills-kg7 (epic), skills-pdg, skills-ebl, skills-bww
|
||||
|
||||
** Commands Used
|
||||
#+begin_src bash
|
||||
# Check AT-SPI status
|
||||
gsettings get org.gnome.desktop.interface toolkit-accessibility
|
||||
# Result: gsettings not available
|
||||
|
||||
# Check if AT-SPI bus running
|
||||
pgrep -a at-spi
|
||||
dbus-send --session --print-reply --dest=org.freedesktop.DBus /org/freedesktop/DBus org.freedesktop.DBus.ListNames | grep -i access
|
||||
# Result: Nothing running
|
||||
|
||||
# Check environment variables
|
||||
echo "NO_AT_BRIDGE=$NO_AT_BRIDGE" # = 1
|
||||
echo "GTK_A11Y=$GTK_A11Y" # = none
|
||||
|
||||
# Test AT-SPI in nix-shell (failed - bus not connected)
|
||||
nix-shell -p at-spi2-core python312Packages.pyatspi python312Packages.pygobject3 --run 'python3 /tmp/test_atspi.py'
|
||||
|
||||
# Start AT-SPI bus manually (worked, but 0 apps)
|
||||
/nix/store/.../at-spi-bus-launcher &
|
||||
# Output: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
|
||||
# Desktop has 0 accessible apps
|
||||
|
||||
# Check uinput access for ydotool
|
||||
ls -la /dev/uinput
|
||||
# crw-rw---- 1 root uinput 10, 223 Dec 16 11:52 /dev/uinput
|
||||
groups | grep -w input
|
||||
# User in both input and uinput groups - ydotool would work
|
||||
|
||||
# Capture window for vision benchmark
|
||||
niri msg action screenshot-window --id 10 --write-to-disk true
|
||||
|
||||
# Check available input tools
|
||||
which ydotool # not installed
|
||||
which wtype # /etc/profiles/per-user/dan/bin/wtype (keyboard only)
|
||||
#+end_src
|
||||
|
||||
** Architecture Notes
|
||||
*** The Seeing Problem - Layered Model
|
||||
#+begin_example
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ "Seeing" Layers │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 4. Understanding: "What can I click? Where's the submit?" │
|
||||
│ → Vision model + prompt engineering │
|
||||
│ → AT-SPI UI tree (if enabled) │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 3. Pixel capture: Screenshots │
|
||||
│ → niri screenshot-window ✅ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 2. Window geometry: Size, position, monitor │
|
||||
│ → niri windows/focused-window ✅ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 1. Window awareness: What's open, app_id, title │
|
||||
│ → niri windows ✅ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
#+end_example
|
||||
|
||||
*** Wayland Security Model Impact
|
||||
- X11 allowed any app to read keystrokes, inject input, screenshot anything
|
||||
- Wayland blocks all of this by design (security feature)
|
||||
- Workarounds require:
|
||||
- Compositor-specific IPC (niri msg)
|
||||
- Kernel bypass for input (ydotool → uinput)
|
||||
- App opt-in for UI tree (AT-SPI)
|
||||
- Portal permissions for screenshots
|
||||
|
||||
*** NixOS AT-SPI Configuration
|
||||
#+begin_src nix
|
||||
# To enable AT-SPI
|
||||
services.gnome.at-spi2-core.enable = true;
|
||||
# For Qt apps also need:
|
||||
# environment.variables.QT_LINUX_ACCESSIBILITY_ALWAYS_ON = "1";
|
||||
#+end_src
|
||||
|
||||
* Process and Workflow
|
||||
|
||||
** What Worked Well
|
||||
- Beads workflow for tracking exploration as issues
|
||||
- Incremental investigation - check what exists before planning
|
||||
- Framing the problem clearly ("seeing problem") helped structure thinking
|
||||
- Quick vision benchmark experiment provided concrete data
|
||||
|
||||
** What Was Challenging
|
||||
- AT-SPI complexity - session vs on-demand, app registration timing
|
||||
- NixOS options discovery - no easy way to search for accessibility options
|
||||
- bd sync worktree error - had to fall back to manual git
|
||||
|
||||
* Learning and Insights
|
||||
|
||||
** Technical Insights
|
||||
- AT-SPI is fundamentally session-level - apps register at startup, can't retroactively add
|
||||
- ~NO_AT_BRIDGE=1~ and ~GTK_A11Y=none~ are NixOS defaults when AT-SPI not enabled
|
||||
- ydotool uses kernel uinput, completely separate from AT-SPI
|
||||
- niri IPC provides rich window metadata including exact pixel dimensions
|
||||
- wtype exists for keyboard input on Wayland but no mouse equivalent installed
|
||||
|
||||
** Process Insights
|
||||
- "OCR is redundant with vision models" - good insight from user that reframed AT-SPI's value
|
||||
- AT-SPI value is semantic structure + precise coordinates, not text extraction
|
||||
- Benchmark-driven approach better than picking one path upfront
|
||||
|
||||
** Architectural Insights
|
||||
- Desktop automation on Wayland requires multiple independent pieces:
|
||||
- Compositor IPC (window management)
|
||||
- Kernel uinput (input injection)
|
||||
- D-Bus accessibility (UI tree)
|
||||
- Portal protocols (screenshots, permissions)
|
||||
- No unified "Playwright for desktop" exists because each piece has different requirements
|
||||
|
||||
* Context for Future Work
|
||||
|
||||
** Open Questions
|
||||
- What coordinate precision can vision models achieve? (benchmark incomplete - need clicking)
|
||||
- Which apps actually expose useful AT-SPI data?
|
||||
- What's the runtime overhead of AT-SPI in practice?
|
||||
- Should AT-SPI be boot-time toggle or always-available?
|
||||
|
||||
** Next Steps
|
||||
- [ ] Create ydotool setup task (enables clicking to verify vision coordinates)
|
||||
- [ ] Enable AT-SPI in NixOS config (when ready to test)
|
||||
- [ ] Complete vision model benchmark with actual coordinate verification
|
||||
- [ ] Document AT-SPI coverage per app (Firefox, Ghostty, etc.)
|
||||
|
||||
** Related Work
|
||||
- [[file:2025-11-08-invisible-window-capture-niri.org][2025-11-08: Invisible Window Capture]] - niri screenshot capability
|
||||
- [[file:2025-11-08-screenshot-analysis-over-engineering-discovery.org][2025-11-08: Screenshot Analysis]] - earlier wayland capture research
|
||||
- skills-kg7: Desktop automation for Wayland/niri (epic)
|
||||
- skills-pdg: Enable AT-SPI for UI tree access
|
||||
- skills-ebl: Benchmark vision model UI understanding
|
||||
- skills-bww: Benchmark AT-SPI overhead and coverage
|
||||
|
||||
* Raw Notes
|
||||
|
||||
** Vision Model Benchmark Experiment
|
||||
Captured Firefox ChatGPT window (1280x1408px). Made coordinate predictions:
|
||||
|
||||
| Element | Predicted X | Predicted Y | Confidence |
|
||||
|---------+-------------+-------------+------------|
|
||||
| "New chat" button | 100 | 176 | High |
|
||||
| "Search chats" | 112 | 218 | High |
|
||||
| "Ask anything" input | 680 | 500 | Medium |
|
||||
| "Thinking" dropdown | 465 | 555 | Medium |
|
||||
| Microphone icon | 927 | 555 | Medium |
|
||||
| Black voice button | 979 | 555 | High |
|
||||
| "+" button (attach) | 377 | 555 | Medium |
|
||||
| User profile | 90 | 981 | High |
|
||||
|
||||
Could not verify via clicking - no mouse input tool available.
|
||||
|
||||
** niri IPC Data Available
|
||||
#+begin_src json
|
||||
{
|
||||
"id": 5,
|
||||
"title": "✳ Boot-time Option",
|
||||
"app_id": "com.mitchellh.ghostty",
|
||||
"pid": 11854,
|
||||
"workspace_id": 3,
|
||||
"is_focused": true,
|
||||
"layout": {
|
||||
"tile_size": [1280.0, 1408.0],
|
||||
"window_size": [1280, 1408]
|
||||
}
|
||||
}
|
||||
#+end_src
|
||||
|
||||
Also available: multi-monitor info with logical positions, scale factors, modes.
|
||||
|
||||
** AT-SPI Test Script
|
||||
#+begin_src python
|
||||
import gi
|
||||
gi.require_version("Atspi", "2.0")
|
||||
from gi.repository import Atspi
|
||||
|
||||
desktop = Atspi.get_desktop(0)
|
||||
count = desktop.get_child_count()
|
||||
print(f"Desktop has {count} accessible apps")
|
||||
|
||||
for i in range(count):
|
||||
app = desktop.get_child_at_index(i)
|
||||
if app:
|
||||
name = app.get_name() or "(unnamed)"
|
||||
print(f" - {name}")
|
||||
#+end_src
|
||||
|
||||
* Session Metrics
|
||||
- Commits made: 4
|
||||
- Files touched: 4 (2 orch docs, 2 beads files)
|
||||
- Lines added/removed: ~+1536/-25 (mostly beads JSON)
|
||||
- Tests added: 0
|
||||
- Tests passing: N/A
|
||||
- Beads issues created: 4 (1 epic, 3 tasks)
|
||||
- Beads issues closed: 1 (skills-d87)
|
||||
Loading…
Reference in a new issue