Google ARTEMIS: test your Android journeys before release
Google ARTEMIS: test your Android journeys before release
Your app opens, the screens look right and the order button is easy to find. Can a customer actually sign in, find a product and enter a delivery address without getting stuck? For a small business delivering across Paris, a break between those steps can cost an order. ARTEMIS, the project hosted at google/artemis, gives an AI assistant hands on Android so it can attempt the journey rather than just comment on a screenshot.
The strongest starting point is acceptance testing: checking that an application meets agreed expectations before release or after a change. That does not require handing over your everyday business phone. It requires a bounded test and an account of what happened: the expected outcome, the observed screen and any checks still requiring a person.
What ARTEMIS lets an assistant do on Android
The official google/artemis repository describes a tool that lets AI assistants and test suites use phones like a human. It can read an interface, tap controls, fill in fields and move between apps. It works with an Android device with authorised USB debugging enabled, or an emulator: a simulated device.
The documentation includes connections to Claude Code, Codex and Antigravity through MCP, a protocol for connecting assistants to tools. This is not a claim that any ordinary Claude conversation can suddenly operate your mobile. A configured connection and device access are necessary. The assistant determines what to attempt; ARTEMIS provides the interaction tools.
That makes it different from Cua Spaces, which gives an agent a separate computer. The question here is whether an Android journey survives a mobile keyboard, permission requests and unexpected pop-ups. A successful desktop browser test cannot settle that question.
Start with a delivery order that will never be placed
Consider a fictional delivery app preparing for a release. You want to know whether a customer can find a product and prepare an order. Using a test account and fictional details, the scenario could be: sign in, search for the product, add it to the basket, enter a test address inside the delivery area and stop before submitting the order.
Write the expected outcome before running the test. The basket should retain the right product, the address should be accepted and the delivery charge should appear. “Check that the app works” leaves the assistant to decide what success means. Your actual business concern might be an address rejected without a useful explanation.
Now imagine an observation: after address entry, the keyboard covers the button needed to continue. This is an illustrative example, not a result from a SAY Digital client test. A screenshot of the blocked screen, the app version and the phone model give a developer something to reproduce. If the journey succeeds, keep evidence of the basket and displayed charge without treating that as proof that payment works.
The deliverable is more useful than a general assurance. You can request a specific fix and repeat the same journey against the next version. The gain comes from comparing outcomes, not from counting how many taps the agent managed.
Specify verification, not just execution speed
Those outcomes depend partly on the execution profile. The README distinguishes Flash, a fast observe-and-act loop, from Pro, which adds planning and more checks. Flash has no checkpoint verification or built-in final report. An ARTEMIS run should not automatically be presented as a documented audit.
Pro offers a component called Checker and configurable final verification; verification can also be switched off. A written report is optional. Ask the delivery team which checks will be enabled and what evidence you will receive. A component named “Checker” cannot establish that your delivery rule was correctly specified in the first place.
App behaviour, permissions, blocked screens and the surrounding context also affect reliability. If the delivery test encounters a request to access contacts, the agreed response should be to record the interruption and preserve the screen, not grant access to keep moving. The team can then decide whether the app is asking for an unnecessary permission or the scenario needs changing.
A reachable button is not permission to press it
Pro also describes a technical check before actions, intended to check the target against the current interface. This is not human approval. Correctly identifying “Place order” does not authorise a purchase. The same boundary applies to publishing content or sending a message to a customer.
In the delivery example, stopping before submission is part of the specification. A named person reviews the evidence and decides what happens next. A demo account reduces the consequences of mistakes, but does not replace a list of prohibited actions. If a sensitive step really needs testing, use an appropriate environment and explicit authorisation rather than letting the task expand midway through execution.
Check the data flows and the full cost
A phone screen may expose names, addresses and messages. The README says the small helper service installed on the phone operates locally and sends nothing elsewhere. That statement concerns the helper, not the complete ARTEMIS workflow. Depending on configuration, screenshots and their context may go to remote AI services for interpretation. Review those flows before introducing customer data.
A dedicated phone, a test account and fictional records make the first trial easier to contain. ARTEMIS uses the Apache 2.0 licence, but an open-source licence does not make the operation entirely free. AI services, any hosting, scenario preparation and failure analysis belong in the budget. Measure the time needed to reach a useful release decision, including investigation and reruns, rather than the apparent cost of a single execution.
Choose one Android journey and one release decision
The project reports a completion rate above 99% on AndroidWorld, a benchmark spanning more than twenty apps and one hundred multi-step tasks. This is the project’s reported result, not a SAY Digital field validation or a guarantee across your phones and workflows. iOS expansion appears on the roadmap; it is not part of the currently described scope.
A practical first trial starts with an Android journey that matters to your customers, a written expected outcome and a person responsible for approval. Ask for comparable evidence and a clear list of what remains untested. Bring that journey to SAY Digital to scope a first test: your team should leave knowing which correction to request and what evidence supports the release decision.
