Building Flows

<!-- TIER:1 -->

Building Flows

A flow moves data from one or more source systems to one or more destination systems. It runs on a schedule, in response to events (webhooks, listeners), or when triggered by another flow. Flows are the primary way integrations get work done in Celigo.

A flow has page generators (exports that fetch data) and page processors (imports and lookups that process each record). Processors run sequentially in a flat list, or conditionally through routers that branch records to different paths. These processing pipeline mechanics -- routers, branches, page processors, response mapping -- are shared with APIs and tools (see building-apis and building-tools).

What Starts a Flow

Flows start themselves -- this is the biggest thing that separates them from APIs (invoked by an HTTP caller) and tools (invoked by a consumer). Every flow begins with one or more page generators, of two kinds:

• Scheduled exports -- the flow runs on a cron cadence and each run pulls from the source: everything (full sync), only what changed since the last run (delta sync), records matching a query, or files landed in an FTP/SFTP/S3 folder. The right primitive for batch work: nightly reconciliations, hourly delta syncs, backfills, off-peak windows.
• Listeners -- the source pushes to the flow. A webhook fires (or a NetSuite/Salesforce native real-time event triggers) and the payload immediately starts flowing. No schedule; the flow runs as events arrive. The right primitive for event-driven work ("when X happens, do Y"), especially when latency matters.

A flow can mix both, and multi-generator designs are common:

• Real-time plus batch safety net -- a listener catches events as they fire; a scheduled export reconciles at off-peak hours, catching up after webhook outages
• Consolidating sources -- customers from Salesforce AND HubSpot, each with its own generator, feeding the same downstream pipeline
• Different slices of the same source -- one export pulls new records, another pulls updated records, when the API exposes them separately

If the requirement is "every night at 2 AM, do X" or "when a webhook arrives, do Y" -- that lives on a flow. APIs and tools have no schedule and no listener; they only run when invoked.

Fetched Data Needs a Downstream Consumer

A common design mistake: ending a flow on an import that fetches data back from a remote system (a preview call, a query, a lookup-shaped POST) and relying on response mapping to capture the result. Response mapping makes fields visible to the NEXT step -- if no next step exists, the captured data is discarded when the run ends and nobody sees it.

When the requirement says "preview / estimate / retrieve / fetch / check / look up", the design needs at least one of:

• A write-back import to the source system (most common) -- e.g. source export -> preview import -> update import that writes the captured fields onto the source record
• A persistent destination the user named (file to S3/SFTP, email, database)
• A router or AI agent step that consumes the captured data within the same run

A two-step export -> fetch-shaped-import flow with nothing after it is a smell -- re-read the intent for where the fetched data should end up. The same applies in reverse: capturing a created record's ID via response mapping is only useful if a later step writes it somewhere.

Flow Topologies

Linear Flows

A flat pageProcessors[] list with no routers. One or more page generators feed records through a sequential chain of page processors. Each processor is either an import (type: "import") or a lookup export (type: "export"). Records pass through every step in order. Unique to flows -- APIs and tools always use routers.

Branching Flows

Page generators feed records into routers[] instead of pageProcessors[]. Each router evaluates records against branch conditions and routes them to matching branches. Branches contain their own pageProcessors[] and can chain to other routers via nextRouterId.

Two routing modes (shared with APIs and tools):

• Input filters (routeRecordsUsing: "input_filters") -- S-expression rules on each branch; last branch can omit filter as a catch-all
• Script-based (routeRecordsUsing: "script") -- a JavaScript function returns the branch name

Flows support both first_matching_branch and all_matching_branches routing. APIs only support first_matching_branch. Tools support first_matching_branch only.

A flow uses EITHER pageProcessors (linear) OR routers (branching) at the top level -- not both.

When a branching flow needs linear steps before the branch point (e.g., a lookup enrichment or AI classification that all branches depend on), use a pass-through router: a single-branch router with nextRouterId pointing to the branching router. Omit routeRecordsTo and routeRecordsUsing on the pass-through router -- including them makes it appear as a filter-based branch in the UI. The API defaults are sufficient.

Abstract / Instance Flows

A template/inheritance model. An abstract flow (isAbstract: true) defines the complete graph but cannot execute. Instance flows (_abstractFlowId) inherit the graph and customize via an overrides object (connections, schedules, mappings, filters).

Use when the same flow structure is deployed across multiple regions, tenants, or environments with different connections or parameters.

Quick Reference

Flow Type Decision Matrix

Minimum Required Fields

Every flow needs at minimum:

• name -- display name
• _integrationId -- parent integration
• disabled: true -- always create disabled
• pageGenerators[] -- at least one entry with _exportId
• Either pageProcessors[] (linear) or routers[] (branching) -- never both

Which Schemas to Read

Always read:

• request.yml -- base flow fields
• page-generator.yml -- export sources, per-generator schedules, delta coordination
• page-processor.yml -- import/export steps with responseMapping and hooks

Add for branching flows:

• router.yml -- routing strategy, record distribution mode
• branch.yml -- input filters, per-branch processors, chaining

Add if response mapping is needed:

• response-mapping.yml -- extract/generate pairs for carrying data between steps

All available schemas (in references/schemas/):

• Base fields (all flows): request.yml
• Response shape: response.yml
• Page generators: page-generator.yml
• Page processors: page-processor.yml
• Response mapping: response-mapping.yml
• Routers: router.yml
• Branches: branch.yml
• Abstract flow helpers: overrides-helper.yml
• Instance overrides: overrides.yml