APM Django - как соотнести журналы контейнеров из Filebeat с данными из ElasticAPM в Kibana?

Версия Kibana:
7.14.0
Elasticsearch версия:
7.14.0
APM Server version:
7.14.0
Filebeat версия:
7.14.0
Язык и версия агента APM Agent:
Python Django - elastic-apm 6.3.3

Описание:
. Сервер APM и агенты Python работают как ожидалось, так же как и Filebeat. Они собирают журналы и метрики, но я не знаю, как соотнести их в Kibana:

Прямая трансляция из раздела Logs работает, и я могу фильтровать экземпляры apm с помощью container.id.

1. Конфигурации APM, Elasticsearch, Kibana

docker-compose.yml:

version: '3.3'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
    hostname: elasticsearch
    environment:
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
      - ELASTIC_PASSWORD=password
    ports:
      - 192.168.100.100:9200:9200
    volumes:
      - ./data:/usr/share/elasticsearch/data
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    networks:
      - elk

  kibana:
    image: docker.elastic.co/kibana/kibana:7.14.0
    hostname: kibana
    restart: always
    ports:
      - 192.168.100.100:5601:5601
    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml:ro
    networks:
      - elk

  apm:
    image: docker.elastic.co/apm/apm-server:7.14.0
    hostname: apm
    command: --strict.perms=false
    depends_on:
      - elasticsearch
    cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
    cap_drop: ["ALL"]
    volumes:
      - ./apm-server.yml:/usr/share/apm-server/apm-server.yml
    ports:
      - 192.168.100.100:8200:8200
    networks:
      - elk

networks:
  elk:
    driver: bridge

apm-server.yml:

apm-server:
  host: "apm:8200"
  secret_token: token
  rum:
    enabled: true
  kibana:
    enabled: true
    host: "kibana:5601"
    protocol: "http"
    username: "elastic"
    password: "password"
setup.template.enabled: true
setup.template.name: "apm-%{[observer.version]}"
setup.template.pattern: "apm-%{[observer.version]}-*"
setup.template.fields: "${path.config}/fields.yml"
setup.template.overwrite: false
setup.template.settings:
  index:
    number_of_shards: 1
    number_of_replicas: 0
    codec: best_compression
    number_of_routing_shards: 30
    mapping.total_fields.limit: 2000
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: elastic
  password: password
  index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
  indices:
    - index: "apm-%{[observer.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"
    - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"
    - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"
    - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"
    - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"
    - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"
monitoring.enabled: true
monitoring.elasticsearch:
  username: "elastic"
  password: "password"
  hosts: ["elasticsearch:9200"]

elasticsearch.yml:

---
cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.type: single-node
xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true

kibana.yml:

---
server.name: kibana
server.host: "0"
server.publicBaseUrl: https://kibana.domain.com
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.username: elastic
elasticsearch.password: password

2. Конфигурация агента Python:

base.py:

(...)
import elasticapm
import logging
from elasticapm.handlers.logging import Formatter
from elasticapm.handlers.logging import LoggingFilter
console = logging.StreamHandler()
console.addFilter(LoggingFilter())
logging.getLogger("").addHandler(console)
fh = logging.FileHandler("spam.log")
formatter = Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)
formatstring = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
formatstring = (
    formatstring + " | elasticapm "
    "transaction.id=%(elasticapm_transaction_id)s "
    "trace.id=%(elasticapm_trace_id)s "
    "span.id=%(elasticapm_span_id)s"
)
ELASTIC_APM = {
    "SERVICE_NAME": "app",
    "SECRET_TOKEN": "secret",
    "SERVER_URL": "http://192.168.100.100:8200",
    "ENVIRONMENT": "test",
}
(...)
MIDDLEWARE = [
    "elasticapm.contrib.django.middleware.TracingMiddleware",
    "elasticapm.contrib.django.middleware.Catch404Middleware",
    (...)
]
INSTALLED_APPS = [
    "elasticapm.contrib.django",
    (...)
]
(...)

3. Конфигурации агента Filebeat

/etc/filebeat/filebeat.yml:

filebeat.inputs:
- type: container
  paths: 
    - '/var/lib/docker/containers/*/*.log'
processors:
- add_docker_metadata:
    host: "unix:///var/run/docker.sock"
- decode_json_fields:
    fields: ["message"]
    target: "json"
    overwrite_keys: true
output.elasticsearch:
  hosts: ["192.168.100.100:9200"]
  indices:
    - index: "docker-%{[agent.version]}-%{+yyyy.MM.dd}"
  username: "elastic"
  password: "password"
logging.json: true
logging.metrics.enabled: false

/usr/share/filebeat/module/elasticsearch/server/ingest/pipeline-json.yml:

(...)
- grok:
    field: message
    patterns: %{GREEDYDATA:msg} | elasticapm transaction.id=%{DATA:transaction.id} trace.id=%{DATA:trace.id} span.id=%{DATA:span.id}
(...)

---

Как заставить его работать? Что я делаю неправильно?

Вернуться на верх