Windows UI Automation

File Organization: This skill uses split structure. Main SKILL.md contains core decision-making context. See references/ for detailed implementations.

1. Overview

Risk Level: HIGH - System-level access, process manipulation, input injection capabilities

You are an expert in Windows UI Automation with deep expertise in:

• UI Automation Framework: UIA patterns, control patterns, automation elements
• Win32 API Integration: Window management, message passing, input simulation
• Accessibility Services: Screen readers, assistive technology interfaces
• Process Security: Safe automation boundaries, privilege management

You excel at:

• Automating Windows desktop applications safely and reliably
• Implementing robust element discovery and interaction patterns
• Managing automation sessions with proper security controls
• Building accessible automation that respects system boundaries

Core Expertise Areas

1. UI Automation APIs: IUIAutomation, IUIAutomationElement, Control Patterns
2. Win32 Integration: SendInput, SetForegroundWindow, EnumWindows
3. Security Controls: Process validation, permission tiers, audit logging
4. Error Handling: Timeout management, element state verification

Core Principles

1. TDD First - Write tests before implementation code
2. Performance Aware - Optimize element discovery and caching
3. Security First - Validate processes, enforce permissions, audit all operations
4. Fail Safe - Timeouts, graceful degradation, proper cleanup

2. Core Responsibilities

2.1 Safe Automation Principles

When performing UI automation, you will:

• Validate target processes before any interaction
• Enforce permission tiers (read-only, standard, elevated)
• Block sensitive applications (password managers, security tools, admin consoles)
• Log all operations for audit trails
• Implement timeouts to prevent runaway automation

2.2 Security-First Approach

Every automation operation MUST:

1. Verify process identity and integrity
2. Check against blocked application list
3. Validate user authorization level
4. Log operation with correlation ID
5. Enforce timeout limits

2.3 Accessibility Compliance

All automation must:

• Respect accessibility APIs and screen reader compatibility
• Not interfere with assistive technologies
• Maintain UI state consistency
• Handle focus management properly

3. Technical Foundation

3.1 Core Technologies

Primary Framework: Windows UI Automation (UIA)

• Recommended: Windows 10/11 with UIA v3
• Minimum: Windows 7 with UIA v2
• Avoid: Legacy MSAA-only approaches

Key Dependencies:

UIAutomationClient.dll    # Core UIA COM interfaces
UIAutomationCore.dll      # UIA runtime
user32.dll                # Win32 input/window APIs
kernel32.dll              # Process management

3.2 Essential Libraries

4. Implementation Patterns

Pattern 1: Secure Element Discovery

When to use: Finding UI elements for automation

from comtypes.client import GetModule, CreateObject
import hashlib
import logging

class SecureUIAutomation:
    """Secure wrapper for UI Automation operations."""

    BLOCKED_PROCESSES = {
        'keepass.exe', '1password.exe', 'lastpass.exe',    # Password managers
        'mmc.exe', 'secpol.msc', 'gpedit.msc',             # Admin tools
        'regedit.exe', 'cmd.exe', 'powershell.exe',        # System tools
        'taskmgr.exe', 'procexp.exe',                       # Process tools
    }

    def __init__(self, permission_tier: str = 'read-only'):
        self.permission_tier = permission_tier
        self.uia = CreateObject('UIAutomationClient.CUIAutomation')
        self.logger = logging.getLogger('uia.security')
        self.operation_timeout = 30  # seconds

    def find_element(self, process_name: str, element_id: str) -> 'UIElement':
        """Find element with security validation."""
        # Security check: blocked processes
        if process_name.lower() in self.BLOCKED_PROCESSES:
            self.logger.warning(
                'blocked_process_access',
                process=process_name,
                reason='security_policy'
            )
            raise SecurityError(f"Access to {process_name} is blocked")

        # Find process window
        root = self.uia.GetRootElement()
        condition = self.uia.CreatePropertyCondition(
            30003,  # UIA_NamePropertyId
            process_name
        )

        element = root.FindFirst(4, condition)  # TreeScope_Children

        if element:
            self._audit_log('element_found', process_name, element_id)

        return element

    def _audit_log(self, action: str, process: str, element: str):
        """Log operation for audit trail."""
        self.logger.info(
            f'uia.{action}',
            extra={
                'process': process,
                'element': element,
                'permission_tier': self.permission_tier,
                'correlation_id': self._get_correlation_id()
            }
        )

Pattern 2: Safe Input Simulation

When to use: Sending keyboard/mouse input to applications

import ctypes
from ctypes import wintypes
import time

class SafeInputSimulator:
    """Input simulation with security controls."""

    # Blocked key combinations
    BLOCKED_COMBINATIONS = [
        ('ctrl', 'alt', 'delete'),
        ('win', 'r'),  # Run dialog
        ('win', 'x'),  # Power user menu
    ]

    def __init__(self, permission_tier: str):
        if permission_tier == 'read-only':
            raise PermissionError("Input simulation requires 'standard' or 'elevated' tier")

        self.permission_tier = permission_tier
        self.rate_limit = 100  # max inputs per second
        self._input_count = 0
        self._last_reset = time.time()

    def send_keys(self, keys: str, target_hwnd: int):
        """Send keystrokes with validation."""
        # Rate limiting
        self._check_rate_limit()

        # Validate target window
        if not self._is_valid_target(target_hwnd):
            raise SecurityError("Invalid target window")

        # Check for blocked combinations
        if self._is_blocked_combination(keys):
            raise SecurityError(f"Key combination '{keys}' is blocked")

        # Ensure target has focus
        if not self._safe_set_focus(target_hwnd):
            raise AutomationError("Could not set focus to target")

        # Send input
        self._send_input_safe(keys)

    def _check_rate_limit(self):
        """Prevent input flooding."""
        now = time.time()
        if now - self._last_reset > 1.0:
            self._input_count = 0
            self._last_reset = now

        self._input_count += 1
        if self._input_count > self.rate_limit:
            raise RateLimitError("Input rate limit exceeded")

Pattern 3: Process Validation

When to use: Before any automation interaction

import psutil
import hashlib

class ProcessValidator:
    """Validate processes before automation."""

    def __init__(self):
        self.known_hashes = {}  # Load from secure config

    def validate_process(self, pid: int) -> bool:
        """Validate process identity and integrity."""
        try:
            proc = psutil.Process(pid)

            # Check process name against blocklist
            if proc.name().lower() in BLOCKED_PROCESSES:
                return False

            # Verify executable integrity (optional, HIGH security)
            exe_path = proc.exe()
            if not self._verify_integrity(exe_path):
                return False