Cel Programs

cel-programs

When to use

Use this skill when tasks include:

• creating or editing cel.yml.hbs agent stream templates
• configuring data stream manifests for the cel input type
• writing CEL programs with pagination, cursor management, or authentication
• testing or debugging a CEL program locally with mito
• setting up system tests with mock APIs for CEL-based data streams
• prototyping a new CEL-based data stream's collection logic
• any CEL or mito question, regardless of context

When not to use

Do not use this skill as the primary guide for:

• ingest pipeline processor design (ingest-pipelines)
• ECS field mapping (ecs-field-mappings)
• package scaffolding (create-integration)
• system test execution with the Elastic stack (integration-testing → references/system-testing.md)

Mandatory workflow — mock → mito → template

This is not a suggestion. Every CEL program MUST be developed in this order. The subagent must not write cel.yml.hbs until the CEL program has been validated with mito against a running mock. Skipping steps or reordering causes failures that are hard to debug.

Do NOT write more than ~10–15 new lines of CEL before running mito. Build the program incrementally in phases (skeleton → error handling → event mapping → pagination → cursor guard), validating with mito after each phase. Writing a large program in one shot leads to cascading compilation errors that are extremely hard to debug. Follow the phased approach in references/cel-incremental-build.md.

Step 3 detail — translating template vars to mito state: When the future cel.yml.hbs will have a state: block like api_key: {{api_key}} and batch_size: {{batch_size}}, the state.json for mito testing uses the same key names with literal test values:

{
  "url": "http://localhost:8090",
  "api_key": "test-key",
  "batch_size": 50,
  "initial_interval": "24h"
}

This mirrors the runtime state the CEL input would provide. Add cursor to test subsequent-run behavior.

For the full mock-first workflow details, CLI flags, execution model, and quality standards: load references/mito-reference.md.

cel.yml.hbs template anatomy

The cel.yml.hbs file at data_stream/<stream>/agent/stream/cel.yml.hbs is a Handlebars template that renders the final CEL input configuration. It has these sections in order:

interval: {{interval}}
resource.tracer:
  enabled: {{enable_request_tracer}}
  filename: "../../logs/cel/http-request-trace-*.ndjson"
  maxbackups: 5
{{#if proxy_url}}
resource.proxy_url: {{proxy_url}}
{{/if}}
{{#if ssl}}
resource.ssl: {{ssl}}
{{/if}}
{{#if http_client_timeout}}
resource.timeout: {{http_client_timeout}}
{{/if}}
resource.url: <constructed from vars>
state:
  <credentials and pagination config from vars>
redact:
  fields:
    - <sensitive state keys>
max_executions: <number, for heavy pagination>
program: |
  <CEL expression>
tags:
{{#if preserve_original_event}}
  - preserve_original_event
{{/if}}
{{#each tags as |tag|}}
  - {{tag}}
{{/each}}
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
{{#if processors}}
processors:
{{processors}}
{{/if}}

Handlebars patterns

Key template fields

• resource.url — base URL, often constructed from multiple vars (e.g., {{url}}/api/v1/endpoint)
• resource.headers (ga 8.18.1) — static headers the same for every request (Content-Type, Accept, API version headers). Set here rather than in-program when headers never vary. Applied before auth headers.
• state: — block where manifest vars are injected as CEL state; credentials and pagination settings go here
• redact.fields — list state keys containing secrets to redact from debug logs
• max_executions — override default 1000 for integrations with heavy pagination (e.g., 5000)
• program: | — the CEL expression; must be a YAML literal block scalar

Do NOT set data_stream.dataset in integration packages

Integration packages (type: integration) must never include data_stream.dataset in cel.yml.hbs or define a data_stream.dataset manifest var. The framework automatically routes documents to the correct data stream. Setting data_stream.dataset overrides this routing and causes documents to land in the wrong index — typically resulting in "0 hits" during system tests.

Only input-type packages (type: input) use data_stream.dataset because they have no predefined data streams.

Data stream manifest configuration

The data stream manifest.yml defines the CEL input stream and its variables.

Standard vars every CEL stream should include

Auth-specific vars depend on the API (API key, OAuth client_id/secret/token_url, bearer token, etc.).

Declare enable_request_tracer in the data stream manifest, not at the input level. Input-level tracing enables logging for all data streams in the policy.

Package-level vs data-stream-level vars

• Package-level vars in the root manifest.yml under policy_templates[].inputs[].vars: shared across streams (e.g., url, auth credentials)
• Data-stream-level vars in data_stream/<stream>/manifest.yml under streams[].vars: stream-specific (e.g., interval, batch_size, initial_interval)

Scope of the CEL program

The CEL program's responsibility is data collection only:

1. Fetch data from the API endpoint(s)
2. Handle pagination — walk through all pages within a single polling cycle
3. Manage cursor state — store timestamps or page tokens in cursor so the next polling interval resumes where the last one left off, avoiding re-collection of already-fetched events
4. Emit raw events — output {"message": e.encode_json()} for each record

The CEL program does not handle: