Docker Compose 部署监控系统 Prometheus + Grafana + Node Exporter + Cadvisor

前言

Prometheus + Grafana + Node Exporter + Cadvisor是一款开源的服务器系统状态监控平台系统,用于管理和存储各服务器的实时信息数据,方便监控和排查服务器故障.博主经过一段时间的安装,配置,测试也基本满足了个人以及中小企业团队对于日志管理的需求.整合了相对详细的部署和配置教程分享给大家.

本教程使用 docker-compose整合部署,参考 stefanprodan/dockprom.

本文为 Stille 原创文章.经实践,测试,整理发布.如需转载请联系作者获得授权,并注明转载地址.

项目流程

服务端:部署 Prometheus + Grafana到主监控服务器
客户端:部署 Node Exporter + Cadvisor到需要被监控的服务器

常规流程

客户端通过 Node Exporter采集系统状态信息, Cadvisor采集 docker容器信息,并开放端口,由服务端 Prometheus进行抓取,并由 Grafana提供前端展示.

服务器部署 Prometheus + Grafana

拉取代码

<pre class="inline:true decode:1 " >git clone -b master https://github.com/stefanprodan/dockprom prom

1	<pre class="inline:true decode:1 " >git clone -b master https://github.com/stefanprodan/dockprom prom

创建 cadday 密码

<pre class="inline:true decode:1 " >docker run --rm caddy caddy hash-password --plaintext 'youtpasswd'
# 获取 hash 密码
JDJhJDE0JHJmeldKeDB0NjJnY0tjQk5wZXVYaC50QjdSbi9aVlVZeFYyV012UkI2Y

<pre class="inline:true decode:1 " >docker run --rm caddy caddy hash-password --plaintext 'youtpasswd'

# 获取 hash 密码

JDJhJDE0JHJmeldKeDB0NjJnY0tjQk5wZXVYaC50QjdSbi9aVlVZeFYyV012UkI2Y

docker-compose.yml

volumes:

创建数据目录

<pre class="inline:true decode:1 " >cd prom
mkdir prometheus_data grafana_data
# 创建数据目录

<pre class="inline:true decode:1 " >cd prom

mkdir prometheus_data grafana_data

# 创建数据目录

修改 docker-compose.yml中顶级卷配置到指定目录用于存储数据

<pre class="inline:true decode:1 " >volumes:
    prometheus_data:
      driver: local
      driver_opts:
        type: 'none'
        o: 'bind'
        device: '/root/prom/prometheus_data'
    grafana_data:
      driver: local
      driver_opts:
        type: 'none'
        o: 'bind'
        device: '/root/prom/grafana_data'

<pre class="inline:true decode:1 " >volumes:

prometheus_data:

driver: local

driver_opts:

type: 'none'

o: 'bind'

device: '/root/prom/prometheus_data'

grafana_data:

driver: local

driver_opts:

type: 'none'

o: 'bind'

device: '/root/prom/grafana_data'

配置 grafana 密码

yourpasswd修改为你的 grafana登陆密码

<pre class="inline:true decode:1 " >    environment:
      - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-yourpasswd}
      - GF_USERS_ALLOW_SIGN_UP=false

<pre class="inline:true decode:1 " > environment:

- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}

- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-yourpasswd}

- GF_USERS_ALLOW_SIGN_UP=false

配置 caddy 密码

修改上文生成的 caddy hash密码

<pre class="inline:true decode:1 " >    environment:
      - ADMIN_USER=${ADMIN_USER:-admin}
      - ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
      - ADMIN_PASSWORD_HASH=${ADMIN_PASSWORD_HASH:-JDJhJDE0JHJmeldKeDB0NjJnY0tjQk5wZXVYaC50QjdSbi9aVlVZeFYyV012UkI2Y

<pre class="inline:true decode:1 " > environment:

- ADMIN_USER=${ADMIN_USER:-admin}

- ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}

- ADMIN_PASSWORD_HASH=${ADMIN_PASSWORD_HASH:-JDJhJDE0JHJmeldKeDB0NjJnY0tjQk5wZXVYaC50QjdSbi9aVlVZeFYyV012UkI2Y

配置 prometheus.yml

使用 file_sd_configs 配置独立配置文件路径,方便随时修改配置而不需要重启容器,并且方便批量分类管理.

创建独立配置目录

在 ./prometheus创建 jobs目录,创建独立配置文件:

nodeexporter.yml
cadvisor.yml
prometheus.yml
pushgateway.yml

独立配置文件

参考示例 ./prometheus/jobs/nodeexporter.yml

<pre class="inline:true decode:1 " >- targets: 
  - &quot;127.127.127.127:9100&quot;
  labels: 
    hostname: Server
- targets: 
  - &quot;127.127.127.128:9100&quot;
  labels: 
    hostname: Server1
- targets: 
  - &quot;127.127.127.129:9100&quot;
  labels: 
    hostname: Server2

<pre class="inline:true decode:1 " >- targets:

- "127.127.127.127:9100"

labels:

hostname: Server

- targets:

- "127.127.127.128:9100"

labels:

hostname: Server1

- targets:

- "127.127.127.129:9100"

labels:

hostname: Server2

参考示例 ./prometheus/jobs/cadvisor.yml

<pre class="inline:true decode:1 " >- targets: 
  - &quot;127.127.127.127:8080&quot;
  labels: 
    hostname: Server
- targets: 
  - &quot;127.127.127.128:8080&quot;
  labels: 
    hostname: Server1
- targets: 
  - &quot;127.127.127.129:8080&quot;
  labels: 
    hostname: Server2

<pre class="inline:true decode:1 " >- targets:

- "127.127.127.127:8080"

labels:

hostname: Server

- targets:

- "127.127.127.128:8080"

labels:

hostname: Server1

- targets:

- "127.127.127.129:8080"

labels:

hostname: Server2

配置 prometheus.yml

参考修改独立配置文件路径 ./prometheus/prometheus.yml

<pre class="inline:true decode:1 " >scrape_configs:
  - job_name: 'nodeexporter'
    file_sd_configs:
      - files:
        - &quot;/etc/prometheus/jobs/nodeexporter.yml&quot;
        refresh_interval: 5s

  - job_name: 'cadvisor'
    file_sd_configs:
      - files:
        - &quot;/etc/prometheus/jobs/cadvisor.yml&quot;
        refresh_interval: 5s

  - job_name: 'prometheus'
    file_sd_configs:
      - files:
        - &quot;/etc/prometheus/jobs/prometheus.yml&quot;
        refresh_interval: 5s

  - job_name: 'pushgateway'
    file_sd_configs:
      - files:
        - &quot;/etc/prometheus/jobs/pushgateway.yml&quot;
        refresh_interval: 5s

<pre class="inline:true decode:1 " >scrape_configs:

- job_name: 'nodeexporter'

file_sd_configs:

- files:

- "/etc/prometheus/jobs/nodeexporter.yml"

refresh_interval: 5s

- job_name: 'cadvisor'

file_sd_configs:

- files:

- "/etc/prometheus/jobs/cadvisor.yml"

refresh_interval: 5s

- job_name: 'prometheus'

file_sd_configs:

- files:

- "/etc/prometheus/jobs/prometheus.yml"

refresh_interval: 5s

- job_name: 'pushgateway'

file_sd_configs:

- files:

- "/etc/prometheus/jobs/pushgateway.yml"

refresh_interval: 5s

客户端配置

docker-compose.yml

防火墙需开放以下相关端口,注意 nodeexporter 使用 network_mode: host 否则流量监控可能不准.

<pre class="inline:true decode:1 " >version: '3.2'
services:
  nodeexporter:
    image: prom/node-exporter:v1.1.2
    container_name: nodeexporter
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    # ports:
      # - 9100:9100
    environment:
      TZ: Asia/Shanghai
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    labels:
      org.label-schema.group: &quot;monitoring&quot;
    restart: always
    network_mode: host


  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.39.0
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      #- /cgroup:/cgroup:ro #doesn't work on MacOS only for Linux
    ports:
      - 8080:8080
    environment:
      TZ: Asia/Shanghai
    labels:
      org.label-schema.group: &quot;monitoring&quot;
    restart: always