Troubleshooting Flows

<!-- TIER:1 -->

Troubleshooting Flows

A flow is broken when it fails to move data correctly. This skill covers systematic diagnosis: identifying the problem type, isolating the failing step, inspecting errors, and resolving them.

Troubleshooting concerns:

• Job status -- understanding what completed, failed, canceled, and retrying mean for the flow
• Error analysis -- grouping errors by pattern to find root causes instead of reading them one-by-one
• Request/response inspection -- seeing exactly what was sent and returned at each step
• Execution logs -- record-level tracing through every stage of the pipeline
• Retry and resolution -- fixing error data and retrying vs bulk resolving
• Delta/state issues -- lastExportDateTime drift, stuck deltas, re-processing windows

Problem Categories

Total Failure

Job status is failed with 0 successful records. The entire run collapsed before processing any data. Typically numPagesGenerated: 0 (export-level failure) or pages generated but numPagesProcessed: 0 (import-level failure on the first page).

Common causes: connection failure (credentials expired, endpoint down), export query error (invalid SQL, bad saved search ID), missing/deleted resource, permission denied.

Partial Failure

Job status is completed but numError > 0 alongside successful records. Some records failed while others processed normally. Real-world data shows wide variance -- from 2 errors in 8000 successes to 500+ errors in 8000 successes.

Common causes: validation errors on the destination (required fields missing, type mismatches), duplicate key violations, record-level lookup failures, rate limiting on specific batches, data-dependent issues (specific records have bad data).

Empty Run

Job completes successfully with 0 errors AND 0 records processed.

Common causes: wrong resourcePath on the export (extracts from wrong JSON path), delta export with no changes since last run (legitimate), output filter too restrictive (all records filtered out), source query returns no results, webhook export with no inbound events.

Stuck or Long-Running

Job stays in running or retrying status longer than expected.

Common causes: large dataset with no pagination limits, destination system slow to respond, script hook with long-running logic, on-premise agent connectivity issues, rate limiting causing backoff.

Intermittent Failures

Flow sometimes succeeds and sometimes fails with the same configuration.

Common causes: token/session expiry mid-run (long-running flows), rate limiting (varies with concurrent flows), transient network errors, source system maintenance windows.

Error Diagnosis Framework

Classification

When an error occurs, classify it into one of three categories to determine the right action:

A 5xx error not in the transient list (e.g., 501) is still likely transient. A 4xx error not in the investigation list warrants manual review.

Root Cause: Configuration vs Data

Every flow error has one of two root causes:

• Static configuration -- a hardcoded value in the step config is wrong (mapping expression, filter rule, hardcoded field, URI, query, SQL statement). Fix: change the resource configuration via celigo <type> set
• Dynamic data -- the upstream source sent unexpected data (missing required field, wrong type, null where a value is expected, unexpected array/object shape). Fix: add input filtering or validation upstream, or fix the source system

To distinguish: check if the error reproduces with different input records. If the same error occurs for every record, it's configuration. If only some records fail, it's data.

# Check if all records fail (configuration) or only some (data)
celigo flows error-summary <flowId>              # Compare error count vs total records
celigo flows errors <flowId> <stepId>            # Sample specific errors to compare

Which Step Failed?

Error location determines which resource and skill to investigate:

Quick Reference

Symptom --> First Command

Key Diagnostic Commands

# Job status
celigo jobs list --flow <flowId> --limit 1           # Most recent job
celigo jobs current --flow <flowId>        # Currently running job
celigo jobs diagnostics <jobId>            # Full diagnostic bundle

# Error investigation
celigo flows error-summary <flowId>        # Per-step error counts
celigo flows error-analysis <flowId> <id>   # Group errors by pattern
celigo flows errors <flowId> <id>          # List individual errors (each has an errorId)
celigo flows error <flowId> <id> <errorId> --request-detail  # Raw HTTP request/response for one error

# Safe iteration first
celigo flows test-run <flowId> --export <genId>  # safe, fast, try this first

# End-to-end execution logging (real run, full per-record I/O)
celigo flows enable-execution-logs <flowId>      # arm debug logging, then run the flow
celigo flows execution-logs <flowId> <jobId>     # list captured per-record logs
celigo flows debug-requests <flowId> <id>        # per-bubble HTTP request/response

Related Skills

• building-flows > Quick Reference -- flow structure, topologies, and configuration
• configuring-exports > Quick Reference -- export configuration and adaptor types
• configuring-imports > Quick Reference -- import configuration and adaptor types