Raspberry Pi 4 as monitoring host

June 10, 2020 4 minutes read

prometheus • Raspberry Pi • victoriametrics

Last weekend I got the new Raspberry Pi 4 with 8G of RAM and to be honest was fascinated how easy it can be configured - I don’t see the real difference between arm64 and amd64 platforms.

Main R Pi’s purpose is NAS & media storage, but I also configured the monitoring stack based on my favorite Prometheus and Victoria Metrics.

For my setup, I attached an external USB drive and configured the ZFS pool on it, installed docker with ZFS driver.
ZFS pool is encrypted (key prompt, i.e. it’s not mounted automatically with the system setup and docker is switched off from the auto start too).
ZFS ARC limited to 3GB (from 8G).

Setup is stable, average CPU temp 40-41C with 55C under full load.

Even in this setup Victoria Metrics noticeably faster on queries from Grafana than Prometheus. And 3 times more efficient on space:

du -sh /storage/monitoring/data/*
308K	/storage/monitoring/data/grafana
68M	/storage/monitoring/data/prometheus
24M	/storage/monitoring/data/vm

vm_info_raspberry vm_active_series_raspberry vm_datapoints_raspberry vm_memory_usage_raspberry

Can Raspberry Pi 4 be used as monitoring server? Definitely.

site_monitoring_raspberry monitoring_probe_raspberry

Setup step by step:

create new ZFS pool with encryption:

apt install zfs-dkms
echo "options zfs zfs_arc_max=3221225472" >> /etc/modprobe.d/zfs.conf
echo 3221225472 > /sys/module/zfs/parameters/zfs_arc_max
zpool create -o ashift=12 \ 
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
-O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \ 
-O atime=off -O mountpoint=/storage storage /dev/sda

To mount the ZFS pool after the OS restart execute the following:

zfs load-key -a
zfs mount -a

install docker with ZFS support:

curl -sSL https://get.docker.com | sh
service docker stop
rm -rf /var/lib/docker/*
zfs create storage/containers
zfs set mountpath=/var/lib/docker storage/containers
cat >> /etc/docker/daemon.json.test<< EOF
{
  "storage-driver": "zfs"
}
EOF
update-rc.d -f docker remove
service docker start

install docker-compose:

sudo curl -L "https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

create the new docker-compose.yml with the monitoring configuration (inspired by rpi_monitoring_with_prometheus):

version: "3.4"
x-extra_hosts:
  &pi
  - "pi:192.168.1.254"
services:
  prometheus:
    image: prom/prometheus:latest
    restart: always
    container_name: prometheus
    extra_hosts: *pi
    privileged: true
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring_config/prometheus.yml:/etc/prometheus/prometheus.yml
      - /storage/monitoring/data/prometheus:/prometheus

  grafana:
    image: grafana/grafana:latest
    restart: always
    container_name: grafana
    extra_hosts: *pi
    depends_on:
      - prometheus
    ports:
      - "3000:3000"
    volumes:
      - /storage/monitoring/data/grafana:/var/lib/grafana

  blackbox_exporter:
    image: prom/blackbox-exporter:master
    restart: always
    container_name: blackbox_exporter
    extra_hosts: *pi
    ports:
      - "9115:9115"
    volumes:
      - ./monitoring_config/blackbox_config.yml:/etc/blackbox_exporter/config.yml

  rpi_cpu_stats:
    image: thundermagic/rpi_cpu_stats:latest
    restart: always
    container_name: rpi_cpu_stats
    ports:
      - "9669:9669"
    environment:
      - PUID=1001
      - PGID=1001
    volumes:
      - /sys:/sys

  victoria-metrics:
    image: victoriametrics/victoria-metrics:latest
    container_name: victoria-metrics
    ports:
      - "8428:8428"
    command: -storageDataPath=/victoria-metrics-data -memory.allowedPercent=30 -retentionPeriod=12
    volumes:
      - /storage/monitoring/data/vm:/victoria-metrics-data

create Prometheus and blackbox exporter configuration in the monitoring_config, examples in the repo.
Docker images are multiplatform and this docker-compose configuration works as well on usual 64 bit systems (amd64).

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# Write to VictoriaMetrics
remote_write:
  - url: http://victoria-metrics:8428/api/v1/write

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['192.168.1.254:9100']

  - job_name: 'pi_cpu_stats'
    static_configs:
      - targets: ['192.168.1.254:9669']

  - job_name: 'victoria_metrics_stats'
    static_configs:
      - targets: ['192.168.1.254:8428']

  - job_name: 'blackbox'
    metrics_path: /probe
    scrape_interval: 30s
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
          - https://stas.starikevich.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.1.254:9115  # The blackbox exporter's real hostname:port.

  - job_name: 'blackbox_ping'
    metrics_path: /probe
    scrape_interval: 30s
    params:
      module: [icmp]
    static_configs:
      - targets:
          - 8.8.8.8
          - 8.8.4.4
          - 1.0.0.1
          - 1.1.1.1
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.1.254:9115  # The blackbox exporter's real hostname:port.

start the monitoring stack:

docker-compose up -d