Prometheus Pushgateway 설치

 

PushGateway ?

서버의 일괄처리 작업(batch job)은 시간 단위나 일 단위 등의 방식으로 정기적인 일정에 따라 수행된다. 이와 같이 일괄처리 작업은 시작되고, 무언가 작업을 수행 한뒤, 종료된다. 지속적으로 동작하지 않기 때문에, 프로메테우스는 이러한 작업에 대한 정보를 정확하게 수집할수 없다. 그렇게 때문에 푸시게이트웨이가 필요하다.

푸시게이트웨이는 서비스 레벨의 일괄처리 작업에 대한 메트릭 캐시다.

 

https://prometheus.io/download/

Download | Prometheus

 

# pushgateway 다운로드 및 설치 

# pwd
/etc/prometheus

# wget https://github.com/prometheus/pushgateway/releases/download/v1.7.0/pushgateway-1.7.0.linux-amd64.tar.gz

# ls -al
-rw-r--r--   1 root       root       10273763 Jan 19 22:30 pushgateway-1.7.0.linux-amd64.tar.gz

# tar -zxvf pushgateway-1.7.0.linux-amd64.tar.gz
pushgateway-1.7.0.linux-amd64/
pushgateway-1.7.0.linux-amd64/LICENSE
pushgateway-1.7.0.linux-amd64/pushgateway
pushgateway-1.7.0.linux-amd64/NOTICE

# ls
alertmanager  console_libraries  consoles  ep-examples-master  prometheus.yml  prometheus.yml.20240225  pushgateway-1.7.0.linux-amd64  pushgateway-1.7.0.linux-amd64.tar.gz

# mv pushgateway-1.7.0.linux-amd64 pushgateway

# rm pushgateway-1.7.0.linux-amd64.tar.gz

 

# 리눅스 서비스 등록

• pushgateway 실행파일 /usr/local/bin/ 복사

# cd etc/prometheus/pushgateway

# ls -al
total 17736
drwxr-xr-x 2       1001       1002     4096 Jan 19 22:30 .
drwxr-xr-x 7 prometheus prometheus     4096 Mar  9 14:19 ..
-rw-r--r-- 1       1001       1002    11357 Jan 19 22:29 LICENSE
-rw-r--r-- 1       1001       1002      487 Jan 19 22:29 NOTICE
-rwxr-xr-x 1       1001       1002 18135918 Jan 19 22:29 pushgateway

# cp pushgateway /usr/local/bin/

# cd /usr/local/bin/

# chown prometheus:prometheus pushgateway

# ls -al
total 317708
drwxr-xr-x  2 root         root              4096 Mar  9 14:23 .
drwxr-xr-x 10 root         root              4096 Aug 10  2023 ..
-rwxr-xr-x  1 alertmanager alertmanager  37345962 Mar  3 09:49 alertmanager
-rwxr-xr-x  1 root         root          17031320 Feb 25 14:58 docker-compose
-rwxr-xr-x  1         1001         1002  19925095 Nov 13 08:54 node_exporter
-rwxr-xr-x  1 prometheus   prometheus   119902884 Sep 30 06:13 prometheus
-rwxr-xr-x  1 prometheus   prometheus   112964537 Sep 30 06:15 promtool
-rwxr-xr-x  1 prometheus   prometheus    18135918 Mar  9 14:23 pushgateway

• pushgateway.service 파일 생성

# cd /etc/systemd/system

# vi pushgateway.service   # (아래 내용 추가)

[Unit]
Description=Push_Gateway
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/pushgateway

[Install]
WantedBy=multi-user.target

• service 확인

# systemctl daemon-reload

# systemctl enable pushgateway
Created symlink /etc/systemd/system/multi-user.target.wants/pushgateway.service → /etc/systemd/system/pushgateway.service.

# systemctl start pushgateway

# systemctl status pushgateway
● pushgateway.service - Push_Gateway
     Loaded: loaded (/etc/systemd/system/pushgateway.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2024-03-09 14:33:08 KST; 5s ago
   Main PID: 2134 (pushgateway)
      Tasks: 6 (limit: 2219)
     Memory: 4.5M
        CPU: 63ms
     CGroup: /system.slice/pushgateway.service
             └─2134 /usr/local/bin/pushgateway

Mar 09 14:33:08 servidor systemd[1]: Started Push_Gateway.
Mar 09 14:33:08 servidor pushgateway[2134]: ts=2024-03-09T05:33:08.625Z caller=main.go:86 level=info msg="starting pushgateway" version="(version=1.7.0, branch=HEAD, revision=109280c17d29059623c6f5dbf1d6babab34166cf)"
Mar 09 14:33:08 servidor pushgateway[2134]: ts=2024-03-09T05:33:08.626Z caller=main.go:87 level=info build_context="(go=go1.21.6, platform=linux/amd64, user=root@c05cb3457dcb, date=20240119-13:28:37, tags=unknown)"
Mar 09 14:33:08 servidor pushgateway[2134]: ts=2024-03-09T05:33:08.642Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9091
Mar 09 14:33:08 servidor pushgateway[2134]: ts=2024-03-09T05:33:08.642Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9091

# netstat -ntpa |grep LISTEN
tcp6       0      0 :::9091                 :::*                    LISTEN      2134/pushgateway

• 웹접속 확인 192.168.56.128:9091

 

prometheus 서버 prometheus.yml 파일에 등록함

 

# cd /etc/prometheus

# pwd
/etc/prometheus

• 아래 소스는 본인 prometheus.yml 샘플 파일이며  scrape_configs: 아래에 아래 내용만 추가

- job_name: pushgateway
  honor_labels: true
  static_configs:
    - targets: ['192.168.56.128:9091']

# vi prometheus.yml

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 192.168.56.128:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "/etc/prometheus/alertmanager/rules/test_rule.yml"
  - "/etc/prometheus/alertmanager/rules/alert_rules.yml"
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.


scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: "node_exporter"
    static_configs:
      - targets: ["192.168.56.128:9100"]
      - targets: ["192.168.56.130:9100"]

  - job_name: 'PostgreSQL_exporter'
    static_configs:
      - targets: ['192.168.56.130:9187', '192.168.56.128:9187']
        #      - targets: ['192.168.56.128:9187']

  - job_name: 'jmx_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.56.130:8081']

  - job_name: 'kubernetes_exporter'
    static_configs:
      - targets: ['192.168.56.10:9100']
      - targets: ['192.168.56.101:9100']
      - targets: ['192.168.56.102:9100']
      - targets: ['192.168.56.103:9100']

  - job_name: 'example'
    static_configs:
      - targets: ['192.168.56.128:8000']

  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['192.168.56.128:9091']

• prometeus 서버 pushgateway 동작 확인

 

 

# pushgateway 저장 확인

• pushgateway 테스트 python sample 파일작성 및 실행

# cat 4-12-pushgateway.py  (python 실행파일 생성 vi로 편집)

from prometheus_client import CollectorRegistry, Gauge, pushadd_to_gateway

registry = CollectorRegistry()
duration = Gauge('my_job_duration_seconds',
        'Duration of my batch job in seconds', registry=registry)
try:
    with duration.time():
        # Your code here.
        pass

    # This only runs if there wasn't an exception.
    g = Gauge('my_job_last_success_seconds',
            'Last time my batch job successfully finished', registry=registry)
    g.set_to_current_time()
finally:
    pushadd_to_gateway('192.168.56.128:9091', job='batch', registry=registry)
    
    
# python3 4-12-pushgateway.py  (python 실행)

• 동작확인