ELK Security: Setup, Configuration, and Best Practices

What is ELK Security?

The term "ELK Security" encompasses two major concepts: using the ELK stack (Elasticsearch, Logstash, Kibana) as a security monitoring and analytics platform, and securing the ELK stack itself against unauthorized access and data breaches. In the context of cybersecurity, the ELK stack serves as a powerful open-source SIEM (Security Information and Event Management) solution capable of aggregating, processing, and visualizing security-related events from across an entire IT infrastructure.

Elasticsearch is a distributed search and analytics engine that stores and indexes large volumes of data. Logstash is a server‑side data processing pipeline that ingests logs, transforms them, and forwards them to Elasticsearch. Kibana provides a web interface for querying, visualizing, and managing the data stored in Elasticsearch. Together they form a robust platform for real‑time threat detection, incident response, and compliance reporting.

Core Components in a Security Context

Elasticsearch – Stores security events as documents; enables fast full‑text search and aggregations to identify anomalies.
Logstash – Parses raw logs (syslog, firewall, web server, endpoint, etc.) into structured fields using filters like grok, mutate, and geoip.
Kibana – Creates dashboards, maps, and alerts to visualize attack patterns and monitor security posture.
Beats (optional but common) – Lightweight shippers such as Filebeat, Auditbeat, Packetbeat, and Winlogbeat that collect and forward data to Logstash or Elasticsearch directly.

Why ELK Security Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Modern IT environments generate massive streams of logs from servers, network devices, applications, and security tools. Without a centralized platform, correlating events to detect advanced threats becomes nearly impossible. ELK Security provides:

Centralized Log Management – Consolidates logs from all sources into a single searchable repository, eliminating blind spots.
Real‑Time Threat Detection – By applying filters, aggregations, and machine learning (via Elasticsearch’s anomaly detection), teams can identify suspicious activities as they happen.
Incident Investigation – Kibana’s discover interface allows analysts to pivot across terabytes of data in seconds, trace attacker movements, and reconstruct timelines.
Compliance and Audit – Satisfies regulatory requirements (PCI DSS, HIPAA, GDPR) by providing immutable log archives and audit trails.
Cost Efficiency – As an open‑source stack, ELK offers a budget‑friendly alternative to commercial SIEMs while remaining highly scalable.

Setting Up ELK for Security Monitoring

This section walks through a basic setup of Elasticsearch, Logstash, and Kibana on a Linux server using Docker containers. The same principles apply to direct package installations or Kubernetes deployments.

Prerequisites

A Linux host with Docker and Docker Compose installed.
Sufficient RAM (at least 4GB for a minimal security lab; 8GB+ recommended).
Open ports: 9200 (Elasticsearch), 5601 (Kibana), and optionally 5044 (Logstash Beats input).

Docker Compose Configuration

Create a file named docker-compose.yml:

version: '3.7'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=MySecurePass123!
    ports:
      - "9200:9200"
    volumes:
      - esdata:/usr/share/elasticsearch/data

  logstash:
    image: docker.elastic.co/logstash/logstash:8.12.0
    container_name: logstash
    ports:
      - "5044:5044"
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=MyKibanaPass456!
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

volumes:
  esdata:

This configuration enables basic security (X‑Pack) with a password for Elasticsearch. For production, you should enable TLS and use proper secrets management.

Starting the Stack

docker-compose up -d

Wait for all services to become healthy. Then access Kibana at http://localhost:5601 and log in with the kibana_system credentials.

Configuring Logstash for Security Logs

Logstash is the heart of log enrichment. A well‑tuned pipeline transforms unstructured text into actionable fields that can be searched and alerted upon.

Example Logstash Configuration

Below is a sample logstash.conf that ingests syslog messages, parses auth logs, and enriches with GeoIP data. Save this file alongside your Docker Compose.

input {
  beats {
    port => 5044
  }
}

filter {
  if [message] =~ /^<[0-9]+>/ {  # syslog priority
    grok {
      match => { "message" => "<%{INT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{INT:pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
    if [syslog_message] =~ /Failed password/ {
      grok {
        match => { "syslog_message" => "Failed password for %{USER:auth_user} from %{IP:source_ip} port %{INT:source_port} %{WORD:protocol}" }
      }
      geoip {
        source => "source_ip"
        target => "geoip"
      }
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://elasticsearch:9200"]
    user => "elastic"
    password => "${ELASTIC_PASSWORD}"
    ssl => true
    ssl_certificate_verification => false
    index => "security-%{+YYYY.MM.dd}"
  }
}

This pipeline:

Accepts data from Beats on port 5044.
Extracts syslog fields using grok patterns.
Specifically identifies failed SSH login attempts, extracting the user, source IP, port, and protocol.
Adds GeoIP location information based on the source IP.
Sends the enriched events to a daily index named security-YYYY.MM.dd.

Setting Up Filebeat to Ship Logs

On your monitored servers, install Filebeat and configure it to forward auth logs to Logstash.

# filebeat.yml (on Linux endpoint)
filebeat.inputs:
  - type: filestream
    paths:
      - /var/log/auth.log
      - /var/log/syslog
    fields:
      log_type: system

output.logstash:
  hosts: ["logstash-host:5044"]

Start Filebeat and verify that events appear in Kibana under the security-* index pattern.

Securing the ELK Stack

A security monitoring platform must itself be hardened. Below are essential steps to protect Elasticsearch, Logstash, and Kibana from unauthorized access and data exfiltration.

1. Enable TLS Encryption

All inter‑node communication and client connections should be encrypted. Elasticsearch supports TLS certificates out of the box. Generate certificates using the elasticsearch-certutil tool or Let’s Encrypt, then configure:

# elasticsearch.yml
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: elastic-certificates.p12

2. Enable Authentication and Role‑Based Access Control (RBAC)

X‑Pack security provides built‑in realms (native, file, LDAP, Active Directory). Create roles and users to enforce least privilege. For example, create a user with read‑only access to security indices:

# In Kibana Dev Tools or via API
POST /_security/role/security_reader
{
  "cluster": [],
  "indices": [
    {
      "names": [ "security-*" ],
      "privileges": [ "read", "view_index_metadata" ]
    }
  ]
}

POST /_security/user/analyst
{
  "password" : "SecureAnalystPass789",
  "roles" : [ "security_reader" ]
}

Assign the analyst user to security team members so they can investigate logs without modifying index configurations.

3. Kibana Spaces and Dashboard Permissions

Kibana Spaces isolate dashboards, visualizations, and saved searches. Create a space for security operations and assign roles with access only to that space:

POST /_security/role/secops_kibana
{
  "cluster": [],
  "indices": [
    {
      "names": [ "security-*" ],
      "privileges": [ "read" ]
    }
  ],
  "applications": [
    {
      "application": "kibana",
      "privileges": [ "feature_dashboard.read", "feature_discover.read" ],
      "resources": [ "space:secops" ]
    }
  ]
}

4. Network Segmentation and Firewall Rules

Expose only Kibana and Logstash (if needed) to the internal network. Elasticsearch should never be directly accessible from untrusted networks. Use firewall rules (iptables, AWS security groups) to restrict traffic:

# Example: Allow only Logstash and Kibana to reach Elasticsearch
iptables -A INPUT -p tcp --dport 9200 -s logstash_ip -j ACCEPT
iptables -A INPUT -p tcp --dport 9200 -s kibana_ip -j ACCEPT
iptables -A INPUT -p tcp --dport 9200 -j DROP

Best Practices for ELK Security

Beyond basic setup and hardening, adhering to operational best practices ensures your security monitoring remains effective and resilient.

Data Lifecycle Management and Index Retention

Security logs can grow exponentially. Implement Index Lifecycle Management (ILM) to automatically roll over, shrink, and delete indices based on age or size. A typical ILM policy for security indices:

PUT _ilm/policy/security_logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Attach the policy to an index template so all new security-* indices follow it automatically.

Alerting and Automation

Proactive detection requires alerts. Use Elasticsearch Watcher (included in X‑Pack) or the open‑source tool ElastAlert to trigger notifications when suspicious patterns emerge. Example Watcher alert for repeated failed logins:

PUT _watcher/watch/failed_logins_watch
{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [ "security-*" ],
        "body": {
          "query": {
            "bool": {
              "must": [
                { "match": { "syslog_message": "Failed password" } }
              ],
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "now-5m"
                  }
                }
              }
            }
          },
          "aggregations": {
            "by_user": {
              "terms": { "field": "auth_user", "size": 100 }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.by_user.buckets.0.doc_count": { "gte": 10 }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "to": "soc@example.com",
        "subject": "Possible brute force attack detected",
        "body": "User {{ctx.payload.aggregations.by_user.buckets.0.key}} has {{ctx.payload.aggregations.by_user.buckets.0.doc_count}} failed logins in 5 minutes."
      }
    }
  }
}

Regularly Update and Patch

Elasticsearch and its plugins receive security updates frequently. Subscribe to the Elastic advisory mailing list and apply patches promptly. Use official Docker images or package repositories to simplify upgrades.

Backup and Disaster Recovery

Security logs are critical evidence. Regularly snapshot your Elasticsearch data using the snapshot lifecycle management (SLM) feature to a remote repository (S3, GCS, or NFS).

PUT _slm/policy/nightly_backup
{
  "name": "",
  "schedule": "0 30 2 * * ?",
  "repository": "my_backup_repo",
  "config": {
    "indices": [ "security-*", ".kibana*" ]
  },
  "retention": {
    "expire_age": "30d",
    "max_count": 10
  }
}

Monitor the ELK Stack Itself

Use Metricbeat and monitoring features to track Elasticsearch cluster health, CPU, memory, and disk usage. Set up dedicated Kibana monitoring dashboards to detect performance degradation before it impacts log ingestion.

Apply the Principle of Least Privilege

Never use the superuser account for day‑to‑day operations. Create distinct roles for log shippers (Filebeat, Logstash) with minimal write permissions, and separate roles for analysts, dashboard editors, and administrators. Regularly