Halo, kalau kamu merasa tulisan saya ngebantu kamu, kamu bisa ucapkan terima kasih lewat saweria .

If you feel this website help you, you can donate at saweria .

Perlu ga aplikasi monitoring internal?

Pertanyaan yang biasa diajukan, jawaban versi saya cuma dua:

Kalau monitoring bawaan dari cloud provider di rasa ga mumpuni.
Kalau anda punya resource punya jalanin aplikasi monitoring nya.

Monitoring bawaan dari cloud provider (di sini saya pake DO) instalasi nya mudah, ringan dan cukup informatif. Hanya saja kurangnya, alertnya hanya sekali (tidak repetitif). lingkup monitoring terbatas (disk, RAM, CPU & network).

Punya ga resource nya? aplikasi monitoring ini lumayan makan resource, kalau emang ga perlu atau ga punya server nganggur lebih baik ga usah.

Karena saya perlu monitoring macem-macem, saya pilih dockprom .

Instalasi dockprom

git clone https://github.com/stefanprodan/dockprom
cd dockprom
ADMIN_USER=admin ADMIN_PASSWORD=admin ADMIN_PASSWORD_HASH=JDJhJDE0JE91S1FrN0Z0VEsyWmhrQVpON1VzdHVLSDkyWHdsN0xNbEZYdnNIZm1pb2d1blg4Y09mL0ZP docker-compose up -d
Otomatis akan terinstall beberapa container di server,
- Prometheus (metrics database) http://:9090 -> yang ngumpulin metrics dari exporter
- Prometheus-Pushgateway (push acceptor for ephemeral and batch jobs) http://:9091
- AlertManager (alerts management) http://:9093 -> buat integrasi alert dengan berbagai macam service (slack, telegram, dll)
- Grafana (visualize metrics) http://:3000 -> monitoring visual
- NodeExporter (host metrics collector) -> mengumpulkan data host (CPU, RAM, disk, swap, dll)
- cAdvisor (containers metrics collector) -> mengumpulkan data container
- Caddy (reverse proxy and basic auth provider for prometheus and alertmanager)

Instalasi Metric Collector di Server

Yang biasa pake Zabbix atau DO monitoring, ini ibaratnya nginstall agent di server. Ada beberapa agent/collector yang biasa saya gunakan, nodeexporter, cadvisor, dan mysqld-exporters (mysql metrics collector)

Copy isi dari docker-compose.exporters.yml atau

version: '2.1'

services:

  nodeexporter:
    image: prom/node-exporter:v1.1.2
    container_name: nodeexporter
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    restart: unless-stopped
    network_mode: host
    labels:
      org.label-schema.group: "monitoring"

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.40.0
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /cgroup:/cgroup:ro
    restart: unless-stopped
    network_mode: host
    labels:
      org.label-schema.group: "monitoring"

  mysqldexporter:
    image: prom/mysqld-exporter
    container_name: mysqldexporter
    enviroment:
      - DATA_SOURCE_NAME=exporter:ppkmlepel4@(localhost:3306)/
    restart: unless-stopped
    network_mode: host
    labels:
      org.label-schema.group: "monitoring"

Catatan, untuk monitoring mysql silahkan buat user baru

CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'ppkmlepel4' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';

Masuk ke server yang ingin di monitoring
Jalankan docker-compose up -d. Apabila hanya butuh monitoring host nya saja jalankan docker-compose up nodeexporters -d
Izinkan port untuk bisa di akses dari server monitoring

firewall-cmd --zone=public  --add-rich-rule='rule family="ipv4" source address="192.168.1.2/32"  port port="9100" protocol="tcp"  accept'

atau

sudo ufw allow from 192.168.1.2/32 to any port 9100 comment "allow nodeexporter access from monitoring server"

Menambahkan Server di dockprom

Pastikan server yang sudah di install metric collector/agent sudah bisa kita akses via server monitoring curl 192.168.1.3:9100/metrics
Masuk ke file prometheus.yml, vim prometheus/prometheus.yml pada bagian target di job_name: exporters tambahkan

scrape_configs:
  - job_name: 'nodeexporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['nodeexporter:9100', '192.168.1.3:9100']

Restart prometheus docker restart prometheus.

Menambahkan Macam-Macam Alert Rules

Semua alert saya dapatkan di sini Awesome Prometheus alerts , alert bawaan dockprom malah ga saya gunakan…

Biar gampang kita pisah-pisahkan saja ya alertnya. Masuk ke folder prometheus di dockprom, kita backup dulu aja alert bawaan si dockprom

mv alert.rules alert.rules.original

Kita buat alert baru, kita namakan host.rules, vim host.rules, isikan dengan


groups:
- name: linux_hosts
  rules:
  - alert: HostOutOfDiskSpace
    expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of disk space (instance {{ $labels.instance }})
      description: "Disk is almost full (< 10% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: HostOutOfMemory
    expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of memory (instance {{ $labels.instance }})
      description: "Node memory is filling up (< 10% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Untuk mysql rules, vim mysql.rules

groups:
- name: mysql
  rules:

  - alert: MysqlDown
    expr: mysql_up == 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: MySQL down (instance {{ $labels.instance }})
      description: "MySQL instance is down on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: MysqlTooManyConnections(>80%)
    expr: avg by (instance) (rate(mysql_global_status_threads_connected[1m])) / avg by (instance) (mysql_global_variables_max_connections) * 100 > 80
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: MySQL too many connections (> 80%) (instance {{ $labels.instance }})
      description: "More than 80% of MySQL connections are in use on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: MysqlHighThreadsRunning
    expr: avg by (instance) (rate(mysql_global_status_threads_running[1m])) / avg by (instance) (mysql_global_variables_max_connections) * 100 > 60
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: MySQL high threads running (instance {{ $labels.instance }})
      description: "More than 60% of MySQL connections are in running state on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: MysqlSlowQueries
    expr: increase(mysql_global_status_slow_queries[1m]) > 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: MySQL slow queries (instance {{ $labels.instance }})
      description: "MySQL server mysql has some new slow query.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: MysqlRestarted
    expr: mysql_global_status_uptime < 60
    for: 0m
    labels:
      severity: info
    annotations:
      summary: MySQL restarted (instance {{ $labels.instance }})
      description: "MySQL has just been restarted, less than one minute ago on {{ $labels.instance }}.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Begitu juga seterus nya ya, utk rule container buat rule baru, misal container.rules lalu copy alert dari Awesome Prometheus alerts ke .rules file
Restart prometheus docker restart prometheus.

Grafana dan Alert Notifications

Untuk grafana tinggal cari dashboard yang cocok di grafana dashboard , dan untuk alert nya silahkan cari sendiri, good luck!

Monitoring Server dengan dockprom