AI-Driven Testing Without a Mac: An Android-First Take

AI-driven testing for mobile apps got another entrant today: Deltix (app.deltix.ai) posted to Hacker News asking for feedback on a platform that drives your app the way a confused user would, then tells you where it fell over.

I read the launch page and the whole thread. The product is interesting. The comment section is more interesting, because three separate people had already built the same thing themselves, on hardware they owned.

---

🔍 What Deltix actually ships today

Stripping the pitch down to what the page states:

Mode	Question it answers	What it does
Task	"Can my user actually do this?"	Ad-hoc run against a new flow
Playbook	"Did we break it?"	Saves a successful run as a deterministic test, replayed each build
Experiment	"Which design wins?"	Same task run against two builds side by side

Setup is four steps: sign up, install the Mac Agent, connect an iOS Simulator, write a task in plain English. The agent navigates and captures "screenshots and the run record" for replay. It is in open beta, free, no card. Their privacy claim is the good part: source and builds stay on your Mac, the agent runs locally against your simulator. You can also bring your own model key so inference bills land on you instead of them.

Physical devices, Android, a CI/CD CLI, and React Native / Flutter support are all listed as coming, not shipped.

One commenter, shermozle, was blunt: "Okay this isn't half baked, it's basically raw. The app opens with everything greyed out. The support email bounces."

That is a day-one beta review, not a verdict. But it matters for how you should plan around this.

---

🍎 The Mac tax nobody in the thread mentioned

Here is the part that decides whether this tool is relevant to you at all. Today it is macOS plus an iOS simulator. For most small teams I know here, that is the whole conversation.

Requirement	Deltix today	DIY Android agent
Host OS	macOS only	Linux, Windows, or macOS
Target platform	iOS simulator	Android emulator or real phone
Hardware floor	A Mac	A laptop you already have, plus optionally a used phone
Physical devices	Listed as coming	Works now over USB
CI integration	CLI listed as coming	Whatever you script
Inference cost	Theirs or your key	Always your key

Most Sri Lankan product teams ship Android first, because that is what the market actually holds. If you are a student, a two-person startup, or a freelancer testing a client's app, you are not buying a Mac to run a beta QA agent.

Key takeaway: The agent is not the moat. ADB is free, the Android emulator is free, and the loop is short. What you are really buying from a hosted product is the harness — replay, run records, CI wiring, device fleet. Judge it on that, not on the demo.

---

🛠️ Building the same loop yourself

The Hacker News thread proves the point better than I can. iamcoder18 wrote: "Just did this locally by giving my AI agent full ADB access to a test phone I have. It's actually kinda amazing just watching it swipe, open the app launcher, and find issues to fix later." jeffnash described their own version built on Android accessibility services. alightsoul said their employer, an airline, already runs Claude over QA workflows through Azure DevOps.

The primitives are all standard Android tooling:

1. See the screen. adb exec-out screencap -p > frame.png for pixels, or adb shell uiautomator dump for the view hierarchy as XML.
2. Send the model both. The XML gives you element bounds and labels; the screenshot catches what the hierarchy lies about.
3. Act. adb shell input tap <x> <y>, adb shell input swipe, adb shell input text "...".
4. Loop until the task is done or a step budget runs out.
5. Record every screenshot, action, and model response so a failing run is reproducible.

# one turn of the loop
adb shell uiautomator dump /sdcard/ui.xml
adb pull /sdcard/ui.xml ./ui.xml
adb exec-out screencap -p > ./frame.png
# → send ui.xml + frame.png + the task to your model
# → model returns: {"action":"tap","x":540,"y":1710}
adb shell input tap 540 1710

The XML-plus-screenshot combination is what jeffnash was circling. Hierarchy alone breaks on canvas-drawn UIs and games. Pixels alone waste vision tokens on text the accessibility tree already gave you for free.

---

💰 Working out what a test run costs you

"Bring your own model key" is honest pricing, and it also means the bill is now your engineering problem. Every loop turn sends a screenshot, so vision tokens dominate, not text.

The arithmetic you need before you leave an agent running overnight:

cost per run ≈ turns × (image tokens + prompt tokens + output tokens) × rate

Rough control knobs, in the order I would reach for them:

• Downscale screenshots. Image token count scales with resolution. A 1080×2400 frame is expensive; you rarely need full resolution to find a button.
• Cap turns per task. A stuck agent will happily tap the same dead button forty times.
• Send the XML, skip the image on turns where the hierarchy is unambiguous.
• Cache the system prompt if your provider supports it. The task instructions do not change between turns.

If you want the numbers instead of the formula, our AI vision token cost calculator prices image inputs by resolution, and the AI model comparison tool lines up rates across providers so you can pick a cheap model for navigation and a strong one only for the judgement calls.

---

⚠️ Where AI test agents actually break

Deltix's Playbook mode is the most thoughtful thing on that page, and it is a tell. Once a run succeeds, it gets frozen into a deterministic replay. That is an admission that you do not want a language model in your CI pipeline making fresh decisions on every commit.

That is the right instinct, and it is the design constraint to copy:

• Exploration is where the model belongs. Non-deterministic, run it when you want to find things.
• Regression must be deterministic. Freeze the successful path into fixed coordinates or element selectors and replay it.
• A red CI build must mean something. If a test fails because the model got creative, your team stops trusting the suite inside a week.

The failure mode to fear is not a missed bug. It is a flaky suite that everyone learns to ignore.

Also budget for the boring bits nobody demos: emulator boot flake, permission dialogs, OTP screens, payment sandboxes. alightsoul already flagged that even with Claude wired into their QA flow, manual intervention has not gone away.

---

🎯 What this means for you

If you build iOS on a Mac, Deltix is free during open beta and worth thirty minutes. Go in expecting rough edges, because the thread found plenty.

If you are like most builders here, the useful takeaway is not the product. It is that a working AI test agent is now a weekend project on hardware you already own:

• Android emulator or a spare phone — no licence, no Mac.
• ADB for eyes and hands.
• A model key you already have from another project.
• A step budget and a screenshot downscaler so it cannot burn your credits while you sleep.

Build the exploration loop first and let it wander through your app. Keep the runs it completes, freeze those into plain scripted tests, and let the model stay on the discovery side of the line. That is the split worth stealing from this launch, and it costs nothing to try.