Kibana Anomaly Detection

Elastic ML Anomaly Detection

Expert process for ML anomaly detection: attribute incidents to entities, explain scores and model behavior, diagnose job lifecycle failures, and manage jobs. Read anomaly results from POST /.ml-anomalies-*/_search (Serverless-safe) and job/datafeed state from ML REST APIs. When the user embeds fixture evidence (influencer rows, job stats) in the prompt, apply the judgment below directly — do not re-fetch fields already supplied.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the `elastic` CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Mode selector

When a question spans modes: Investigate → Explain → Troubleshoot. Finish one mode before blending logic.

Serverless note: Legacy /_ml/anomaly_detectors/{job_id}/results/* endpoints return HTTP 410 in Serverless. Always query .ml-anomalies-* via POST /.ml-anomalies-*/_search with result_type filters.

Score quick reference

• record_score bands: >75 critical · 50–75 warning · 25–50 minor · <25 informational
• multi_bucket_impact ≥ 3 → sustained shift (not a transient spike)
• initial_record_score >> record_score → renormalization (model saw worse anomalies later)
• actual << typical with count/low_count/low_mean → absence/outage, not just a low value
• Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity

Full score definitions, renormalization mechanics, and anomaly_score_explanation components: references/score-reference.md.

Core concepts

Treat .ml-anomalies-* as layered result types via result_type in search queries:

Read scores this way:

• anomaly_score / record_score = current normalized values (move as the model sees new extremes).
• initial_anomaly_score / initial_record_score = immutable snapshots from detection time.
• `influencer_score` ranks entity responsibility within a bucket — the highest score is the primary suspect, not the bucket-level anomaly_score alone.
• Map entities via partition_field_value / by_field_value / over_field_value.
• Read multi_bucket_impact (-5 to +5) to separate single-bucket spikes from sustained trends.

Mode: Investigate — RCA

When: "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.

Process

1. Discover jobs. Call GET /_ml/anomaly_detectors when the job ID is unknown. Call GET /_ml/anomaly_detectors/{job_id} and GET /_ml/datafeeds/datafeed-{job_id} to learn source indices, entity fields (by_field_name, over_field_name, partition_field_name), and bucket_span. The decision: identify the related job group — jobs sharing a datafeed index or entity field monitor the same system from different angles.

1. Scope the incident window. Call POST /.ml-anomalies-*/_search with result_type: bucket, a time range, and optional minimum anomaly_score. The decision: fix the incident start/end and count how many jobs co-fire in that window. Low scores across many jobs simultaneously often indicate a systemic root cause.

1. Attribute to entities (critical for RCA). For the anomalous bucket timestamp, call POST /.ml-anomalies-*/_search with result_type: influencer, the job ID(s), and the bucket time range. Sort by `influencer_score` descending. The decision: name the entity with the highest `influencer_score` as the likely cause — it ranks how unusual each entity is in that bucket. Do not restate only the bucket anomaly_score without attributing responsibility. Recommend drilling into that entity's records next.

1. Cross-job confirmation. Re-query influencers (or bucket records) across related job IDs for the same entity values and time window. Entities anomalous in 2+ jobs are prime suspects (resource fault or systemic failure); single-job entities are often downstream victims. See references/protocols/investigation.md.

1. Drill into records. Call POST /.ml-anomalies-*/_search with result_type: record, exact job ID, entity filters (partition_field_value, by_field_value), and low minimum record_score (25 or lower). Read multi_bucket_impact ≥ 3 as sustained behavioral shift. Read actual vs typical for fault class (spike vs absence/outage).

1. Confirm with source evidence. Call POST /{index}/_search on the datafeed source index for the suspect entity and time window. Raw source documents are ground truth — never close an RCA without them.

1. Synthesize. Report: root cause entity · affected jobs · temporal progression · fault class · severity · recommended actions. Worked walkthrough: references/worked-example.md. Query