Bahasa kueri untuk TSDB. Meningkatkan PromQL (Alexander Valyalkin, VictoriaMetrics) (2019)

Selamat sore! Hari ini saya akan bercerita sedikit tentang PromQL. Ini adalah bahasa kueri untuk database deret waktu. Kemudian saya akan memberi tahu Anda bagaimana kami meningkatkannya dalam database VictoriaMetrics yang kami kembangkan deret waktu.

PromQL adalah bahasa kueri yang digunakan dalam sistem pemantauan Prometheus.

PromQL time-series — , SQL, InfluxQL Flux, , Graphite. time-series . , PromQL – time-series , , , time-series .

SQL , time-series .

? node_network_receive_bytes_total. , timestamps . , . , JOIN.

, PromQL. PromQL .

s

PromQL Grafana. rate, total. counter, , . , . , rate , .

PromQL SQL.

, . . SQL , PromQL. SQL — . :

• SQL SELECT , , PromQL, — , , .
• PromQL JOIN.
• GROUP BY. , .
• . .
• HAVING.
• , SQL.

, time-series . SQL — , time-series .

https://github.com/grafana/grafana/issues/11451

https://github.com/prometheus/prometheus/issues/3746

https://github.com/prometheus/prometheus/issues/3806

PromQL – data model Prometheus PromQL. Brian Brazil.

- , , , , Brain , .

• - – , PromQL , , scrape_interval, .. , . , .
• - – , increase(), , , . , 10, increase() 9,5. - , data model Brian Brazil. . .
• PromQL , min/max rate. Rate – , . . . rate , , .
• rate [d]. , , , rate. , PromQL, . , , , , , Prometheus .

PromQL , Grafana, Grafana , .

, Grafana. .

?

- – VictoriaMetrics. time-series , .

• time-series , long-term remote storage Prometheus, . . Prometheus.
• Prometheus, Influx, OpenTSDB, Graphite, CVS, JSON lines, Prometheus text exposition format.
• VictoriaMetrics PromQL, PromQL.
• VictoriaMetrics PromQL MetricsQL. , . .

:

– rate.

PromQL VictoriaMetrics. VictoriaMetrics PromQL, .

? $__interval. Grafana , . , . VictoriaMetrics, . rate — rate(m), rate(m[$__interval]).

. . , , 5. , 5 . PromQL : rate(m[5i]).

. , . . , . , PromQL deriv(), .

, MetricsQL VictoriaMetrics. . , 5 , , 5 . step() . .

VictoriaMetrics -, , , , . , Prometheus , VictoriaMetrics , .

Prometheus rate() , . , , Prometheus . VictoriaMetrics , , , , , scrape_interval time-series , , . . 0,1 . , 1 , , scrape_interval, . . , Prometheus, , -, 10 .

VictoriaMetrics rollup , . , . .

VictoriaMetrics rollup , rollup. , .

rollup_rate. rate. "" (Counters). , — . MetricsQL.

rollup_rate . , , . , , .

rate , . , . . , , , rate, , rate .

VictoriaMetrics range_*, . , .

, , , quantile, sum, , .

? , . - rate . rate range_avg(). , , .

default, , default.

, sinus, , 0,3. sinus. rand, . . . , , , . rand 0, .

label’. PromQL label’. : label_replace label_join, — , , .

MetricsQLL , label’:

• label_set – labels time-series. label bar query.
• label_del – labels query.
• label_keep – labels, , .
• label_copy label_move – label.

https://victoriametrics.com/promql/expand-with-exprs

PromQL , PromQL-, , . WITH templates. common table expressions – CTE. .

– . PromQL-. , {instance=~”$node:$port”, job=~”$job”} .

WITH templates . with (cf = {instance=~”$node:$port”, job=~”$job”}). cf – , . , . , cf.

– WITH templates . ru, : free, limit, filters. .

Ru – resource usage. Free – , . Limit – . Filters – .

, resource usage resource utilization, . . .

WITH , . node exporter labels. . , , , . resource usage.

. WITH templates templates. templates. ru, . .

ru MetricsQL VictoriaMetrics, . . WITH . ru.

– , . – . .

MetricsQL, VictoriaMetrics. , - , - . , , .

.

• if, ifnot.
• PromQL: rate(q[1.5m]) offset 0.5d.
• start() end() , .
• smooth_exponential(), .
• , , .

MetricsQL , template meta-variables Grafana.

, Grafana template variables. template variables . Grafana , variables. , , .

, PromQL Grafana , template variables. , , - - , , .

Template meta_variables . , WITH templates variables Grafana, , template meta-variables, Grafana.

issue pull request. issue, PR. Grafana PR. , , : ?.

template meta-variables. variables, commonFilters. , variables: job, instance.

, , . WITH templates, Prometheus.

• .
• Template meta-variables meta-variables.
• . , .
• Template meta-vars Prometheus, datasource Grafana.

VictoriaMetrics , .

single-node , . , , . . time-series , . open source . Update: — . single-node

WITH templates Grafana?

. . – template meta-variables, WITH templates .

, , ? InfluxQL, Graphite, Prom’?

, . PromQL MetricsQL, . , , - , MetricsQL.

. . Graphite, ? - ?

time-series , VictoriaMetrics , , . . matric name , value.

Graphite .

Graphite .

Graphite .

VictoriaMetrics . . . .

. . - ?

. Graphite metric name VictoriaMetrics, . . . , Influx line protocol, . (tags & fields). metric names. .

!

alerting. Thanos, PromQL , alerting Prometheus . PromQL alerting, ? alerting?

VictoriaMetrics PromQL, Prometheus. Prometheus’, . Grafana, Prometheus, datasource url VictoriaMetrics. , . .

alerting — alerting' VictoriaMetrics. . . , alerting Prometheus, VictoriaMetrics. alerting , Prometheus, VictoriaMetrics, alerting rules Prometheus'. datasource. VictoriaMetrics, alerts, alert manager. Update: — vmalert.

VictoriaMetrics ?

VictoriaMetrics , . , . . , . . , . . . , time-series .

. , PromQL.

.

, .

, .

, JOIN. JOIN ? ?

VictoriaMetrics , Prometheus, Graphite, . . . VictoriaMetrics , Prometheus, , VictoriaMetrics.

. . , ?

, . VictoriaMetrics, Thanos, Prometheus', . VictoriaMetrics . . VictoriaMetrics. VictoriaMetrics , . , - , Thanos. Thanos Prometheus', . Prometheus. VictoriaMetrics .

, !

! , VictoriaMetrics single-node, . open source - - ?

Apache2 . , , , . . . , , , VictoriaMetrics.

! InfluxDB, Timescale. ? - benchmarks?

. InfluxDB . . ClickHouse . Altinity, , ClickHouse, . ClickHouse time-series Influx, -. Timescale . . https://altinity.com/blog/tag/benchmark/ .

, . , , . . , .

VictoriaMetrics , ClickHouse, , ClickHouse, . . : - , - .

?

benchmarks , 100 %. ClickHouse, , 100 % CPU .

, !

, ?

. , VictoriaMetrics, , , , . , . storage, google cloud . , - , . . - . . , , . . . , production, , , . Update: VictoriaMetrics — . https://victoriametrics.github.io/Cluster-VictoriaMetrics.html#replication-and-data-safety .

. - , , , ?

, , PromQL. Offset . offset 1d, . — , , , .

- ? . ., , , , ?

PromQL — quantile_over_time. , Prometheus, , , . , quantile_over_time(0.99, m[24h]) 99- 24 .

!

Seberapa terbuka model pengembangannya? Anda akan terbuka sekarang. Setelah itu, dua opsi dimungkinkan: kami melakukan penggabungan setiap enam bulan sekali, dan mereka ada di sana selama 5 GB, atau Anda mengembangkan secara terbuka

Kami akan berusaha melakukan pembangunan secara terbuka. Mungkin pertanyaannya diparafrasekan secara berbeda? Apakah kami akan menggabungkan permintaan penarikan pihak ketiga?

Ini adalah bagian kedua dari pertanyaan itu.

Kami akan melakukannya, tapi tidak semua. Jika kami dapat melihat bahwa mereka berguna, kami akan melakukannya. Tetapi kami akan mencoba melakukannya secara berbeda dari Brain Brazil, yaitu, kami tidak akan mengatakan bahwa kami memiliki model data dan hanya itu. Jika ada permintaan tarik yang bagus, maka kami akan menggabungkan.