software-mansion/argent4 files

Argent Device Interact

Interact with an iOS simulator, Android emulator, or Chromium (CDP) app using argent MCP tools. Use when tapping UI elements, performing gestures, scrolling/swiping, typing text, pressing hardware buttons, launching apps, opening URLs, taking screenshots, waiting for an element to appear or disappear, or checking visible app state after interactions. Not for TV targets.

Specification
Skill ID
software-mansion/argent/argent-device-interact
Publisher
software-mansion
Repository
argent
Installs
287
Files
4
Synced
Sep 16, 2026
How to use it

Open any RiverX project, open the Skills panel in the chat, and search for this identifier. The files are fetched from the source repository at install time.

software-mansion/argent/argent-device-interactInstalls these files
  • SKILL.md
  • references/chromium.md
  • references/gesture-examples.md
  • references/secrets.md

What this skill tells the agent

Unified tool surface

All interaction tools below accept a udid parameter and auto-dispatch iOS vs Android based on its shape (UUID → iOS simulator, chromium-cdp-<port> → Chromium (CDP) app, anything else → Android adb serial). You use the same tool names on every platform.

Chromium (CDP) app = an Electron app or a Chromium-family browser (Chrome/Brave/Edge) exposing a Chrome DevTools Protocol endpoint. The same describe/tap/keyboard/screenshot surface drives it, but scrolling, tabs, cookies and storage differ — read `references/chromium.md` before driving a `chromium` target.

1. Before You Start

If you delegate simulator tasks to sub-agents, make sure they have MCP permissions.

Use list-devices to get a target id. Results are tagged with platform (ios, android, or chromium); booted/ready devices come first. Pick the first entry that matches the platform you need — if none are ready, call boot-device with udid (iOS), avdName (Android), or electronAppPath (boots an Electron app as a chromium device). A Chromium browser already running with a CDP port shows up directly — no boot-device needed. See argent-ios-simulator-setup / argent-android-emulator-setup for full setup flow.

Load tool schemas before first use. Gesture tools (gesture-tap, gesture-swipe, gesture-pinch, gesture-rotate, gesture-custom) may be deferred — their parameter schemas are not loaded until fetched. Always use ToolSearch to load the schemas of all gesture tools you plan to use before calling any of them. If you skip this step, parameters may be coerced to strings instead of numbers, causing validation errors.

2. Best Practices

  1. Always refer to tapping_rule from your argent.md rule before tapping.
  2. Before performing interactions, consider whether they can be dispatched sequentially - more on that in run-sequence.
  3. Use `gesture-swipe` for lists/scrolling, not gesture-custom, unless you need non-linear movement. On Chromium use gesture-scroll instead — gesture-swipe is touch-only. Consider whether you need multiple swipes, if yes - use run-sequence. Pass momentum: false when the swipe should decelerate before ending for a precise movement.
  4. Tap a text field before typing, then use keyboard to enter text.
  5. Coordinates are normalized — always 0.0–1.0, not pixels.
  6. For app navigation, use the element tree returned after each action (--- Elements after action (describe) ---); call describe only when no fresh tree is available for the current screen. It works on any screen without app restart. Do not navigate from screenshot pixels on regular in-app screens unless the tree failed to expose a reliable target. Use native-describe-screen only when you need app-scoped UIKit properties.

3. Opening Apps

Never navigate to an app by tapping home-screen icons. Use launch-app or open-url — they are instant and reliable.

launch-app — by bundle ID

{ "udid": "<UDID>", "bundleId": "com.apple.MobileSMS" }

Common IDs: com.apple.MobileSMS (Messages), com.apple.mobilesafari (Safari), com.apple.Preferences (Settings), com.apple.Maps, com.apple.Photos, com.apple.mobilemail, com.apple.mobilenotes, com.apple.MobileAddressBook (Contacts)

open-url — by URL scheme

{ "udid": "<UDID>", "url": "messages://" }

Common schemes: messages://, settings://, maps://?q=<query>, tel://<number>, mailto:<address>, https://... (Safari)

4. Choosing the Right Tool

ActionToolNotes
Multiple actionsrun-sequenceBatch steps in one call (no intermediate screenshots)
Open an applaunch-appAlways — never tap home-screen icons
Restart an apprestart-appTerminate and relaunch by bundle ID
Open URL/schemeopen-urlWeb pages, deep links, URL schemes
Single tapgesture-tapButtons, links, checkboxes
Scroll/swipegesture-swipeStraight-line scroll or swipe
Scroll (Chromium)gesture-scrollWheel-based; deltas are window fractions, positive deltaY = down
Drag (Chromium)gesture-dragSliders, drag-and-drop, text selection
Long pressgesture-customContext menus, drag start
Drag & dropgesture-customComplex drag interactions
Pinch/zoomgesture-pinchTwo-finger pinch with auto-interpolation
Rotationgesture-rotateTwo-finger rotation with auto-interpolation
Custom gesturegesture-customArbitrary touch sequences, optional interpolation
Hardware keybuttonHome, back, power, volume, appSwitch, actionButton
Type textkeyboardEvery platform. Text or one named key per call, never both
Paste textpasteOnly where a user would paste (OTP code, long link). Sim/emu only
Rotate devicerotateOrientation changes
Shake deviceshakeShake handlers (sim/emu only), Undo-typing prompt, RN dev menu
Wait for UIawait-ui-elementBlock until an element is visible/hidden/exists/contains text
Wait for idleawait-screen-idleBlock until a non-empty screen tree stops changing

5. Finding Tap Targets

IMPORTANT. When moved to a different screen after an action or do not know the coordinates of component, always perform proper discovery first.

App typeDiscovery toolWhat it returns
Target app discoverydescribeAccessibility element tree for the current device screen (iOS AX-service, Android uiautomator, or Chromium DOM walker) with normalized frame coordinates. Works on any app, system dialogs, and Home screen — no app restart or bundleId required
React Nativedebugger-component-treeReact component tree with names, text, testID, and (tap: x,y)
App-scoped nativenative-describe-screenLow-level app-scoped accessibility elements with normalized and raw coordinates; requires bundleId