Prometheus 与 Grafana 的安装与配置

0x0 学习资源

https://yunlzheng.gitbook.io/prometheus-book/

https://songjiayang.gitbooks.io/prometheus/

0x1 Prometheus 的安装与配置

Docker Compose 方式安装

Copy
version: '3.3'
services:
prometheus-server:
# https://github.com/bitnami/bitnami-docker-prometheus
image: bitnami/prometheus:2.23.0
container_name: prometheus-server
volumes:
# https://prometheus.io/docs/prometheus/latest/configuration/configuration/
- /etc/prometheus/prometheus.yml:/opt/bitnami/prometheus/conf/prometheus.yml
- prometheus_data:/opt/bitnami/prometheus/data
command:
- "--config.file=/opt/bitnami/prometheus/conf/prometheus.yml"
- "--storage.tsdb.path=/opt/bitnami/prometheus/data"
- "--web.console.libraries=/opt/bitnami/prometheus/conf/console_libraries"
- "--web.console.templates=/opt/bitnami/prometheus/conf/consoles"
# https://www.prometheus.io/docs/guides/basic-auth/
- --web.external-url=http://localhost:80/prometheus
- --web.route-prefix=/
ports:
- '9090:9090'
logging:
driver: json-file
options:
max-size: "200k"
max-file: "10"
restart: unless-stop

volumes:
prometheus_data:

反向代理配置

Nginx

Copy
# https://www.prometheus.io/docs/guides/basic-auth/
# https://github.com/prometheus-community/prometheus-playground/tree/master/nginx
location /prometheus {
proxy_pass http://127.0.0.1:9090/;
}

Exporters and Integrations

Exporters and integrations

Chrome Metrics 语法高亮插件

fhemberger/prometheus-formatter

参考资料

https://www.qikqiak.com/post/blackbox-exporter-on-prometheus/

https://www.qikqiak.com/post/prometheus-book/

0x2 Grafana 的安装与配置

Docker Compose 方式安装

Copy
version: '3.3'
services:
grafana-server:
# https://github.com/bitnami/bitnami-docker-grafana
image: bitnami/grafana:7.3.4
container_name: grafana-server
environment:
# https://grafana.com/docs/grafana/latest/administration/configuration/
- 'GF_SECURITY_ADMIN_USER=admin'
- 'GF_SECURITY_ADMIN_PASSWORD=admin_password'
# https://grafana.com/tutorials/run-grafana-behind-a-proxy/
- 'GF_SERVER_HTTP_PORT=8848'
- 'GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/grafana/'
- 'GF_SERVER_SERVE_FROM_SUB_PATH=true'
volumes:
- grafana_data:/opt/bitnami/grafana/data
ports:
- '8848:8848'
logging:
driver: json-file
options:
max-size: "200k"
max-file: "10"
restart: unless-stop

volumes:
grafana_data:

反向代理配置

Nginx

Copy
# https://grafana.com/tutorials/run-grafana-behind-a-proxy/
location /grafana/ {
proxy_pass http://127.0.0.1:8848/;
}

参考资料

https://www.qikqiak.com/post/grafana-usage-in-k8s/

0x3 容器间组网

创建外部容器交换机

Copy
docker network create homelab_net

加入外部容器交换机

Copy
services:
whoami:
...
networks:
- homelab_net

networks:
homelab_net:
external: true

0x4 万物可监控

Exporters and integrations

推荐 Grafana 监控模板

Grafana Dashboards - discover and share dashboards for Grafana.

推荐模板

Docker 容器监控：cAdvisor

使用 Google 开源的 cAdvisor 容器监控

只做实时数据监控，不存储监控数据，配合 Prometheus 效果更佳

参考教程

Copy
VERSION=v0.36.0 # use the latest release version from https://github.com/google/cadvisor/releases
sudo docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
--privileged \
--device=/dev/kmsg \
gcr.io/cadvisor/cadvisor:$VERSION

*NIX 系统监控：Node Exporter

使用 Prometheus 开源的 Node Exporter

参考教程

Copy
docker run -d \
--net="host" \
--pid="host" \
-v "/:/host:ro,rslave" \
quay.io/prometheus/node-exporter \
--path.rootfs=/host

Copy
node-exporter:
image: quay.io/prometheus/node-exporter
restart: always
network_mode: "host"
pid: "host"
ports:
- '9100:9100'
command:
- --path.rootfs=/host
volumes:
- "/:/host:ro,rslave"

网络探测：Blackbox Exporter

使用 Prometheus 开源的 Blackbox Exporter

参考教程

Copy
docker run --rm -d -p 9115:9115 --name blackbox_exporter -v `pwd`:/config prom/blackbox-exporter:master --config.file=/config/blackbox.yml

Nginx 监控：Nginx Prometheus Exporter

Copy
# nginx.conf
location = /metrics {
stub_status;
}

Copy
docker run -p 9113:9113 nginx/nginx-prometheus-exporter:0.8.0 -nginx.scrape-uri http://<nginx>:8080/stub_status

Grafana Dashboard