Skip to content
ios-agent

Development

Terminal window
uv run pytest tests/unit          # 856 tests, no device, no model
uv run pytest tests/tui           # 234 tests, the terminal front end
uv run pytest tests/integration   # 18 simulator + 3 device tests
uv run pytest tests/evals -s      # golden flows, with cost per flow
uv run ruff check . && uv run mypy ios_mcp agent/ios_agent tui/ios_tui

The eval suite is the quality gate: it reports tokens, wall time, action count and resolution-tier distribution per flow. A drift from exact toward text-fuzzy is the leading indicator that a flow is about to become flaky. Agent tasks additionally declare an action floor, the number of actions a hand-written oracle needs, asserted against that oracle so it cannot drift into an aspiration, and a turn floor, the model calls a perfect batcher would need for the same route. The turn floor is derived rather than declared: batch.simulate_turns walks what the oracle actually did and splits it wherever the batch guard would stop, so it measures the guard rather than a number typed beside it. Failures are attributed too: a report says which of them were the device, perception, the model or the policy gate, rather than only that something failed.

Those numbers are kept over time in tests/evals/history.jsonl, one committed line per measured run. CI runs the one series that costs nothing (the oracle against a scripted device) and fails if any of it moves without the new line being committed alongside. Hand-run slices go in the same file:

Terminal window
python scripts/eval_trend.py show --suite agent-oracle
python scripts/eval_trend.py append .artifacts/evals/agent-s5.json --suite agent-model

The guard is exact rather than banded, because every number it checks is a count on a fixed route with no model, no network and no clock in it. See docs/adr/0009.

Terminal window
uv run python scripts/tui_screenshot.py   # render the front end to .artifacts

A passing test suite says nothing about what a terminal app looks like. That script has caught eight display bugs no assertion did.

Three distributions in one uv workspace, and the dependencies only point one way. See ARCHITECTURE.md:

ios-tui    terminal front end     depends on ios-agent, ios-mcp
ios-agent  goal-directed agent    depends on ios-mcp
ios-mcp    library + MCP server   depends on neither