Agentforce Test

ADLC Test

Automated testing for Agentforce agents with smoke tests, batch execution, and iterative fix loops.

Overview

This skill provides comprehensive testing capabilities for Agentforce agents, including automated utterance derivation from agent subagents, preview-based smoke testing, trace analysis, an iterative fix loop for identified issues, and security testing (OWASP LLM Top 10). It bridges the gap between initial development and production deployment.

Security testing is part of the ADLC, not a separate skill. Functional correctness (right topic, right action) and security posture (resists attacks) are two dimensions of the same test suite. Treat adversarial coverage as part of the test flow and the Agent Spec — when you plan tests for an agent, plan its security tests too. Security test-case generation is gated on explicit user confirmation (see Mode C).

Platform Notes

• Shell examples below use bash syntax. On Windows, use PowerShell equivalents or Git Bash.
• Replace python3 with python on Windows.
• Replace /tmp/ with $env:TEMP\ (PowerShell) or %TEMP%\ (cmd).
• Replace jq with python -c "import json,sys; ..." if jq is not installed.
• find ... | head -1 -> Get-ChildItem -Recurse ... | Select-Object -First 1 in PowerShell.

Usage

This skill uses sf agent preview and sf agent test CLI commands directly. There is no standalone Python script.

Quick smoke test (Mode A):

# Start preview, send utterance, end session (--authoring-bundle generates local traces).
# Run from inside the Salesforce project directory (the CLI requires sfdx-project.json).
# With --authoring-bundle, `start` REQUIRES an action mode: --simulate-actions or
# --use-live-actions. The mode flag belongs on `start` only — `send` and `end` reject it.
sf agent preview start --json --authoring-bundle MyAgent --simulate-actions -o <org-alias>
sf agent preview send --json --session-id <ID> --utterance "test" --authoring-bundle MyAgent -o <org-alias>
sf agent preview end --json --session-id <ID> --authoring-bundle MyAgent -o <org-alias>

Batch testing (Mode B):

# Deploy and run test suite
sf agent test create --json --spec test-spec.yaml --api-name MySuite -o <org-alias>
sf agent test run --json --api-name MySuite --wait 10 --result-format json -o <org-alias>

Security testing (Mode C — confirm with the user before generating):

# You read the .agent file and write the security cases yourself — same as
# Mode B, with security-specific guidance in references/security-test-design.md.

# C1: deploy the security suite you authored (identical to Mode B)
sf agent test create --json --spec /tmp/MyAgent-security-spec.yaml --api-name MyAgent_Security -o <org-alias>

# C2: live adversarial probing (identical to Mode A, one fresh session per case).
# --simulate-actions is the C2 default: probe the agent's reasoning without firing
# real Apex/Flow writes. Only substitute --use-live-actions on explicit user opt-in.
sf agent preview start --json --authoring-bundle MyAgent --simulate-actions -o <org-alias>
sf agent preview send --json --session-id <ID> --utterance "<payload>" --authoring-bundle MyAgent -o <org-alias>
sf agent preview end --json --session-id <ID> --authoring-bundle MyAgent -o <org-alias>

Action execution:

# Execute a Flow or Apex action directly via REST API
TOKEN=$(sf org display -o <org-alias> --json | jq -r '.result.accessToken')
INSTANCE_URL=$(sf org display -o <org-alias> --json | jq -r '.result.instanceUrl')
curl -s "$INSTANCE_URL/services/data/v63.0/actions/custom/flow/Get_Order_Status" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"inputs": [{"orderId": "00190000023XXXX"}]}'

Testing Workflow

This skill supports three testing modes plus direct action execution:

• Mode A: Ad-Hoc Preview Testing -- Quick smoke tests during development using sf agent preview. No test suite deployment needed (org authentication still required). Best for iterative development and fix validation.
• Mode B: Testing Center Batch Testing -- Persistent test suites deployed to the org via sf agent test. Best for regression suites, CI/CD, and cross-skill integration with /agentforce-observe.
• Mode C: Security Testing (OWASP LLM Top 10) -- Adversarial testing across 7 OWASP categories. You write the cases yourself by reading the agent's own .agent script and business domain, using the neutral technique catalog in assets/payloads/ as a coverage checklist. Two sub-modes over the same authored case set: C1 deploys them as a Testing Center security suite (AiEvaluationDefinition, mechanically identical to Mode B); C2 probes them live via sf agent preview (mechanically identical to Mode A) with A–F severity grading. Generating security test cases requires explicit user confirmation.
• Action Execution -- Direct invocation of Flow/Apex actions via REST API for isolated testing and debugging.

When to use which:

Mode A: Ad-Hoc Preview Testing

Full reference: references/preview-testing.md

Test Case Planning

If no utterances file is provided, auto-derive test cases from the .agent file:

1. Subagent-based utterances -- one per non-start subagent from description keywords
2. Action-based utterances -- target each key action
3. Guardrail test -- off-topic utterance
4. Multi-turn scenarios -- subagent transitions
5. Safety probes -- adversarial utterances (always included)

Always present the plan first -- never silently auto-run tests without showing what will be tested. Ask the user to review/modify before executing.

Preview Execution

Use --authoring-bundle to compile from the local .agent file (enables local trace files). Run these from the Salesforce project directory; --authoring-bundle requires an action mode on start (--simulate-actions or --use-live-actions), and that flag is valid on start alone:

SESSION_ID=$(sf agent preview start --json \
  --authoring-bundle MyAgent \
  --simulate-actions \
  --target-org <org> 2>/dev/null \
  | jq -r '.result.sessionId')

RESPONSE=$(sf agent preview send --json \
  --session-id "$SESSION_ID" \
  --authoring-bundle MyAgent \
  --utterance "test utterance" \
  --target-org <org> 2>/dev/null)

# Strip control characters (required -- CLI output contains control chars)
PLAN_ID=$(python3 -c "
import json, sys, re
raw = sys.stdin.read()
clean = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f]', '', raw)
d = json.loads(clean)
msgs = d.get('result', {}).get('messages', [])
print(msgs[-1].get('planId', '') if msgs else '')
" <<< "$RESPONSE")

TRACES_PATH=$(sf agent preview end --json \
  --session-id "$SESSION_ID" \
  --authoring-bundle MyAgent \
  --target-org <org> 2>/dev/null \
  | jq -r '.result.tracesPath')

Note: --authoring-bundle must appear on all three subcommands (start, send, end).

Trace Location and Analysis

Traces are written to: .sfdx/agents/{BundleName}/sessions/{sessionId}/traces/{planId}.json

Key trace analysis commands:

# Topic routing
jq -r '.topic' "$TRACE"
jq -r '.plan[] | select(.type == "NodeEntryStateStep") | .data.agent_name' "$TRACE"

# Action invocation
jq -r '.plan[] | select(.type == "BeforeReasoningIterationStep") | .data.action_names[]' "$TRACE"

# Grounding check
jq -r '.plan[] | select(.type == "ReasoningStep") | {category: .category, reason: .reason}' "$TRACE"

# Safety score