Configuring Lookup Caches

<!-- TIER:1 -->

Configuring Lookup Caches

A lookup cache is an in-memory key-value store managed by Celigo for fast data retrieval during integration processing. Unlike export-based lookups that query external systems per-record, lookup caches provide sub-millisecond access to pre-loaded reference data without consuming API calls.

Lookup caches handle three concerns:

• Cache metadata -- name, integration scope, and whether data is included in templates/cloning
• Data management -- loading, retrieving, updating, and purging key-value entries via dedicated data endpoints
• Size governance -- each cache has a 50 MB limit; the per-environment aggregate limit across all caches is 1 GB

Lookup caches are consumed by import and export mappings via the lookups[] array. A lookup entry references a cache by _lookupCacheId and optionally extracts a specific field from the cached object via the extract JSON path. This is configured on the import/export resource, not on the cache itself.

Used across flows, APIs, and tools.

"Lookup" -- Which One Is Meant?

"Lookup" is overloaded in Celigo, and a lookup cache is only one thing the word can mean. When a request just says "the lookup," confirm which one before acting -- they live in different places and are configured differently. A single entry in lookups[] resolves its value one of three ways:

• Cached -- the entry sets _lookupCacheId to read a lookup cache (this skill): pre-loaded reference data read in-memory, no per-record external call, as fresh as the last data load.
• Static map -- the entry carries an inline map object. No cache and no external call; the table travels with the import/export. Best for a handful of stable pairs that rarely change.
• Live / dynamic -- the entry issues a per-record query against a connected system (an HTTP request, SOQL, a NetSuite search, SQL). Always-fresh answers, at the cost of one external request per record.

Quick test for which is meant: pre-loaded data read in-memory is a lookup cache; an answer that must reflect the source system right now is a live/dynamic lookup; a small fixed table living on the step is a static `map`.

The same word also names things outside this skill: a mid-flow lookup step (an export with isLookup: true that fetches from an external system between flow steps) and transform lookups (tables inside an export transform's rules). "Lookup" alone is never enough -- listen for whether the subject is stored cache data, an inline rule on a step, a live query, or a separate flow step.

Use Cases

Lookup caches serve several distinct patterns in production integrations:

Cross-Reference / ID Mapping

The most common use case. Map external IDs to internal IDs across systems (e.g., Shopify product IDs to NetSuite internal IDs, external customer GUIDs to Salesforce Account IDs). Avoids repeated API lookups during import processing.

Deduplication / Debouncing

Track which records have already been processed to prevent duplicate operations. Store record IDs or composite keys as cache keys; check existence before processing.

State Tracking

Maintain timestamps, batch IDs, or processing markers across flow runs. Common patterns: "last sync time" caches, "open batch ID" holders, lock tables for concurrency control.

Static Reference Data

Store relatively stable reference tables -- product catalogs, category hierarchies, shipping overrides, zip code lookups. Set includeDataInTemplatesAndCloning: true when this data is part of the integration's configuration.

Quick Reference

Minimum Required Fields

Every lookup cache needs at minimum: name.

Optional but common: description, _integrationId (scopes the cache to an integration), includeDataInTemplatesAndCloning.

Data Operations

All data operations use the cache ID. Data entries are key-value pairs where both key and value are strings. Values can be JSON strings for complex objects -- the consuming lookup's extract JSON path pulls specific fields from the parsed value.

Schema Index

All schemas are in references/schemas/:

• Base fields (create/update): request.yml -- name, description, includeDataInTemplatesAndCloning, externalId
• Response shape: response.yml -- includes size, sizeInMB, timestamps, integration/connector refs

Referencing Caches in Lookups

Lookup caches are consumed through the lookups[] array on imports and exports. See the writing-mappings skill for full lookup configuration details. The key fields:

• name -- unique identifier for the lookup within the resource
• _lookupCacheId -- references the cache resource
• extract -- optional JSON path to pull a specific field from the cached value (e.g., $.details.price)
• default -- fallback value when the key is not found
• allowFailures -- when true, missing keys use the default instead of failing the record

Reference the lookup by name in Handlebars expressions: {{lookup 'lookupName' record.fieldName}}.

Related Skills

• writing-mappings > Add lookups for value translation -- configuring lookups[] with _lookupCacheId, extract, and allowFailures
• writing-handlebars > Quick Reference -- {{lookup}} helper syntax for referencing caches in templates
• configuring-imports > How to Build an Import -- imports are the primary consumer of lookup caches
• configuring-exports > Quick Reference -- lookup exports (isLookup: true) as an alternative to caches for live data
• building-flows > How to Build a Flow -- wiring lookups into flow pipelines

<!-- TIER:2 -->

How to Build a Lookup Cache

1. Determine the use case

What data needs to be cached? Cross-reference IDs, static reference tables, deduplication keys, or processing state? This determines data structure, loading strategy, and lifecycle.

2. Check for existing caches

Before creating a new cache, search for existing ones in the account:

# List all lookup caches
celigo lookup-caches list

# Search across the account
celigo account search "lookup cache"
celigo account search "<keyword related to your use case>"

3. Create the cache

Create the cache metadata first. Data is loaded separately.

# Create a new empty cache
echo '{"name":"Product SKU to NetSuite ID","description":"Maps Shopify SKUs to NetSuite internal IDs for order import"}' | celigo lookup-caches create

# With integration scope and template inclusion
echo '{"name":"State Shipping Overrides","_integrationId":"<id>","includeDataInTemplatesAndCloning":true}' | celigo lookup-caches create

4. Load data into the cache

Use the put-data command to upsert key-value entries. The CLI auto-batches by count (1000 entries) and size (5 MB) per request.

# Upsert entries
echo '{"data":[{"key":"SKU-001","value":"{\"nsId\":\"12345\",\"name\":\"Widget A\"}"},{"key":"SKU-002","value":"{\"nsId\":\"12346\",\"name\":\"Widget B\"}"}]}' | celigo lookup-caches put-data <cacheId>

# Simple string values for deduplication