---
name: ui-query
description: Query UI elements via AT-SPI accessibility tree. Get window lists, extract text, find elements by role/name without OCR.
tools: scripts/ui-query
---

# UI Query Skill

Query desktop UI elements via AT-SPI (Assistive Technology Service Provider Interface). This provides semantic access to UI data - buttons, labels, text fields, their states and positions - without OCR.

## Prerequisites

AT-SPI must be enabled on the system:
- NixOS: `services.gnome.at-spi2-core.enable = true`
- Qt apps need: `QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1`

Apps must be started after AT-SPI is enabled to register with the bus.

## Commands

### list-windows

List all windows visible to AT-SPI:

```bash
scripts/ui-query list-windows           # Table format
scripts/ui-query list-windows --json    # JSON output
scripts/ui-query list-windows -v        # Include element counts (slower)
```

### get-text

Extract text content from windows:

```bash
scripts/ui-query get-text 'Firefox'      # Window containing 'Firefox'
scripts/ui-query get-text --app waybar   # All windows from app
scripts/ui-query get-text --all          # Everything (verbose)
scripts/ui-query get-text 'Editor' --json
```

### find-element

Find UI elements by role or name:

```bash
scripts/ui-query find-element --role button           # All buttons
scripts/ui-query find-element --name 'Save'           # Elements named 'Save'
scripts/ui-query find-element -r button -n OK         # OK buttons
scripts/ui-query find-element -w Firefox -r link      # Links in Firefox
scripts/ui-query find-element --role text --limit 5   # First 5 text elements
```

### query-state

Get detailed state for specific elements:

```bash
scripts/ui-query query-state --focused            # Current focused element
scripts/ui-query query-state --role checkbox      # All checkboxes with states
scripts/ui-query query-state --name 'Save' -a     # 'Save' elements, all states
```

Shows: states, actions, values, text content, geometry, caret position.

### benchmark

Measure AT-SPI coverage and performance:

```bash
scripts/ui-query benchmark              # Full audit
scripts/ui-query benchmark --app waybar # Single app
scripts/ui-query benchmark --json       # JSON output
```

See `docs/benchmark-results.md` for baseline measurements.

## All Commands Support

- `--json` - JSON output
- `--debug` - Show AT-SPI errors (normally suppressed)

## Supported Applications

| Type | Support |
|------|---------|
| GTK apps | Good - automatic AT-SPI support |
| Qt apps | Good - with QT_LINUX_ACCESSIBILITY_ALWAYS_ON |
| Firefox | Needs explicit enablement in about:config |
| Electron | Varies - often poor support |
| Terminals | Basic - window frames, no buffer text |

## Debugging

Check AT-SPI bus:
```bash
dbus-send --session --dest=org.a11y.Bus --print-reply /org/a11y/bus org.freedesktop.DBus.Peer.Ping
```

Explore accessibility tree visually:
```bash
accerciser
```

## Complementary Skills

- **niri-window-capture**: Visual screenshots (what it looks like)
- **ui-query**: Semantic data (what it contains)

Use together for comprehensive UI understanding.
