Akan ada terjemahan dari 2 posting tentang eksportir grok.

Terjemahan pertama: Cara menggunakan eksportir grok untuk menghasilkan metrik prometheus dari log yang tidak terstruktur

Mari kita bicara tentang eksportir grok. Pada artikel ini, saya akan menjelaskan bagaimana eksportir grok dapat digunakan untuk membuat metrik prometheus dari jurnal yang tidak terstruktur.

Grok ELK (ElasticSearch, Logstash, Kibana) Fabian Stäber grok exporter.

grok exporter => https://github.com/fstab/grok_exporter

1: Grok exporter

zip grok exporter https://github.com/fstab/grok_exporter/releases.

1. ​ (releases) ( v0.2.7).
2. ​ zip-, . — 64- Linux. .

wget https://github.com/fstab/grok_exporter/releases/download/v0.2.7/grok_exporter-0.2.7.linux-amd64.zip

1. ​ .
2. ​ , grok exporter.

[root@localhost grok_exporter-0.2.7.linux-amd64]# ./grok_exporter -config ./config.yml
Starting server on http://localhost.localdomain:9144/metrics

http://localhost.localdomain:9144/metrics.

2:

Grok exporter. , .

30.07.2016 04:33:03 10.3.4.1 user=Nijil message="logged in"
30.07.2016 06:47:03 10.3.4.2 user=Alex message="logged failed"
30.07.2016 06:55:03 10.3.4.2 user=Alex message="logged in"
30.07.2016 07:03:03 10.3.4.3 user=Alan message="logged in"
30.07.2016 07:37:03 10.3.4.1 user=Nijil message="logged out"
30.07.2016 08:47:03 10.3.4.2 user=Alex message="logged out"
30.07.2016 14:34:03 10.3.4.3 user=Alan message="logged out"

, . Prometheus .

1 , , config.xml, grok exporter. .

global:
    config_version: 2
input:
    type: file
    path: ./example/nijil.log  # Specify the location of the your log
    readall: true              # This should be True if you want to read whole log and False if you want to read only new lines.
grok:
    patterns_dir: ./patterns    
metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'

server:
    port: 9144

.

global:
    # Config version
input:
    # How to read log lines (file or stdin).
grok:
    # Available Grok patterns.
metrics:
    # How to map Grok fields to Prometheus metrics.
server:
    # How to expose the metrics via HTTP(S).

3: Grok exporter

, , Prometheus.

metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'

grok – %{SYNTAX:SEMANTIC}, SYNTAX — , , SEMANTIC – . %{HOSTNAME:instance}, HOSTNAME – grok, IP- , IP- ( ), . , SYNTAX , , IP- . , DATE, TIME, HOSTNAME, USER GREEDYDATA , , " " .

, , . , . , (SEMANTIC of the SYNTAX) . . . . (Counter), grok exporter , .

grok exporter ./grok_exporter -config ./config.yml . , user_activity, .

# TYPE user_activity counter
user_activity{user="Alan"} 2
user_activity{user="Alex"} 3
user_activity{user="Nijil"} 2

Prometheus . , prometheus, Prometheus .

: Apache grok exporter

https://www.robustperception.io/getting-metrics-from-apache-logs-using-the-grok-exporter

, .

, , , , . grok. , Apache, access.log:

x.x.x.x - - [20/Jan/2020:06:25:24 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16061 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16064 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET /blog/rss HTTP/1.1" 301 3478 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET / HTTP/1.1" 200 16065 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET /blog/feed HTTP/1.1" 200 3413 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET /feed HTTP/1.1" 200 6496 "-" "Emacs Elfeed 3.2.0"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"

:

wget https://github.com/fstab/grok_exporter/releases/download/v1.0.0.RC2/grok_exporter-1.0.0.RC2.linux-amd64.zip

unzip grok_exporter-*.zip
cd grok_exporter*amd64

:

cat << 'EOF' > config.yml
global:
    config_version: 2
input:
    type: file
    path: access.log
    readall: true
grok:
    patterns_dir: ./patterns
metrics:
    - type: counter
      name: apache_http_response_codes_total
      help: HTTP requests to Apache
      match: '%{COMBINEDAPACHELOG}'
      labels:
          method: '{{.verb}}'
          path: '{{.request}}'
          code: '{{.response}}'
server:
    port: 9144
EOF

grok exporter:

./grok_exporter -config config.yml

http://localhost:9144/metrics :

# HELP apache_http_response_codes_total HTTP requests to Apache
# TYPE apache_http_response_codes_total counter
apache_http_response_codes_total{code="200",method="GET",path="/"} 5
apache_http_response_codes_total{code="200",method="GET",path="/blog/feed"} 1
apache_http_response_codes_total{code="200",method="GET",path="/feed"} 1
apache_http_response_codes_total{code="301",method="GET",path="/blog/rss"} 1

readall, , .

Grok – , Logstash (Logstash — L ELK). , , Apache. Grok , , . COMMMONAPACHELOG , ,

COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

. , . Go ( Prometheus alerting notification) .

, Grok , :

    - type: summary
      name: apache_http_response_bytes
      help: Size of HTTP responses
      match: '%{COMMONAPACHELOG}'
      value: '{{.bytes}}'

, , :

    - type: gauge 
      name: apache_http_last_request_seconds
      help: Timestamp of the last HTTP request
      match: '%{COMMONAPACHELOG}'
      value: '{{timestamp "02/Jan/2006:15:04:05 -0700" .timestamp}}'

(timestamp) grok exporter, time.Parse Golang. (divide), .

, Grok. , , , .