Ecs Field Mappings

ecs-field-mappings

When to use

Use this skill when tasks include:

• adding or modifying files under data_stream/<stream>/fields/
• populating ecs.yml with ECS field references
• selecting event.kind, event.category, event.type, and event.outcome values
• choosing field type and mapping properties (metric_type, dimension, multi_fields, and related options)
• checking whether a field already exists in ECS before adding custom fields
• troubleshooting mapping validation/build failures from elastic-package check, elastic-package lint, or pipeline test schema checks

ECS dependency configuration

Every package needs _dev/build/build.yml at the package root. This file pins the ECS schema version used for field resolution.

dependencies:
  ecs:
    reference: "git@v9.3.0"

This file is required whenever the package has any field file. The scaffold does not generate it — create it manually. If it is missing or uses an outdated version, tests report ECS fields as undefined (e.g., field "destination.ip" is undefined).

Field files and roles

A data stream's fields/ directory contains a small set of YAML files with distinct responsibilities:

base-fields.yml

Fixed routing constants and @timestamp. All six fields are ECS fields, so each entry uses external: ecs. Override type and value where the data stream needs a constant_keyword with a fixed value — the description is inherited from ECS automatically.

- name: data_stream.type
  external: ecs
- name: data_stream.dataset
  external: ecs
- name: data_stream.namespace
  external: ecs
- name: event.module
  external: ecs
  type: constant_keyword
  value: <package_name>
- name: event.dataset
  external: ecs
  type: constant_keyword
  value: <package_name>.<stream_name>
- name: '@timestamp'
  external: ecs

Do not add other fields here. Only these routing constants and @timestamp belong in base-fields.yml.

constant_keyword candidates

Fields that hold a single value for every document in a data stream should use constant_keyword. Beyond the routing constants in base-fields.yml, evaluate these:

When a constant_keyword field is also an ECS field (e.g., observer.vendor), use external: ecs with the type override. This inherits the description from ECS and avoids manual duplication. Place the definition in the appropriate field file (ecs.yml for most ECS fields, base-fields.yml for routing constants):

- name: observer.vendor
  external: ecs
  type: constant_keyword
  value: Acme Corp

`remove_from_source` option: Because constant_keyword stores the value once in index metadata, it does not need to appear in every document's _source. Elasticsearch handles this automatically — no explicit _source.excludes configuration is needed. This saves storage when the value is always the same.

ecs.yml

Populate this file with the ECS fields the pipeline sets that dynamic mapping cannot infer. Use only name and external: ecs for each entry — no type, no description. The type is resolved from the ECS schema via _dev/build/build.yml.

On packages whose conditions.kibana.version floor is >= 8.13.0, the ecs@mappings component template dynamically maps standard keyword/date and standard-prefix geo ECS fields, so explicit declarations are REQUIRED only for types it cannot infer (geo_point on non-standard parent prefixes, geo_shape, nested, flattened) or where elastic-package validation fails. The gate is the kibana version constraint, not the package-spec version — ecs@mappings is applied by the stack at install time from 8.13 onward, and a spec-3.x package that still admits 8.11/8.12 installs does not get it there, so such packages still need every pipeline-set ECS field declared. Declaring more is harmless for a builder, but is never something a reviewer should demand — and existing declarations are not removable-noise findings either (see review-integration/references/conflict-resolutions.md).

When you DO declare a field whose name exists in ECS, external: ecs must be used (wiki reference). This applies across field files — ecs.yml, base-fields.yml, and any file that defines an ECS field. You may override properties (e.g., type: constant_keyword, value:) while still using external: ecs — the description is inherited from ECS. Do not use external: ecs in fields.yml, agent.yml, or beats.yml — those files define non-ECS fields.

- name: event.kind
  external: ecs
- name: event.category
  external: ecs
- name: event.type
  external: ecs
- name: event.outcome
  external: ecs
- name: event.action
  external: ecs
- name: source.ip
  external: ecs
- name: source.port
  external: ecs
- name: destination.ip
  external: ecs
- name: user.name
  external: ecs
- name: related.ip
  external: ecs
- name: related.user
  external: ecs

When attaching extra metadata to an ECS field (for example making a field a TSDB dimension or a constant_keyword with a fixed value), combine external: ecs with that metadata. The description is inherited from ECS. Place the definition in ecs.yml (or base-fields.yml for routing constants):

- name: observer.vendor
  external: ecs
  type: constant_keyword
  value: Acme Corp

fields.yml

Integration-specific custom (non-ECS) fields only. Use a nested group hierarchy for the vendor namespace:

- name: acme.firewall
  type: group
  fields:
    - name: rule_id
      type: keyword
    - name: policy_name
      type: keyword
    - name: bytes_in
      type: long
      unit: byte
      metric_type: gauge

Groups do not need to be declared as type: object — defining a group with nested fields is sufficient. The object structure is implicit.

labels.* exception

labels is a core ECS object (type: object, object_type: keyword) designed for ad-hoc key-value metadata. Subkeys under labels.* do not require vendor namespacing — this is the one exception to the vendor-prefix rule.

Use labels.* for simple keyword flags or integration-internal markers (e.g., labels.is_ioc_transform_source). Use the vendor namespace for structured or nested data from an upstream source.

Flags vs structured data

Boolean flags and simple tags can live flat under the vendor group:

- name: acme.firewall
  type: group
  fields:
    - name: is_encrypted
      type: boolean
    - name: policy_name
      type: keyword

Structured data from the source should use sub-groups for logical hierarchy:

- name: acme.firewall
  type: group
  fields:
    - name: rule
      type: group
      fields:
        - name: id
          type: keyword
        - name: name
          type: keyword
        - name: action
          type: keyword

agent.yml

Non-ECS fields populated by the Elastic Agent or Beats framework but not covered by ECS. Include only when the input type emits these fields. Typical fields: cloud.image.id, cloud.instance.id, host.containerized, host.os.build, host.os.codename, input.type, log.offset.

See references/root-and-core-fields.md for full YAML samples.

beats.yml

Filebeat/Beats-specific fields not covered by ECS. Minimal form contains input.type and log.offset. Some inputs also emit log.flags or log.file.* sub-fields.