Level=info ts=T15:26:51.283Z caller=main.go:302 msg="Starting StatsD -> Prometheus Exporter" version="(version=, branch=, revision=)" $ statsd_exporter -statsd.listen-udp localhost:8125 -log.level debug First, create a Python virtual environment where Airflow will be installed:ġ 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 In this tutorial, I am using Python 3 and Apache Airflow version 1.10.12. Follow me to the next section, where we are going to start by installing Apache Airflow. deploy the Prometheus server to collect the metrics and make them available to Grafanaīy the end of the blog, you should be able to watch the Airflow metrics in the Grafana dashboard.convert the statsd metrics to Prometheus metrics using statsd_exporter.configure Airflow to publish the statsd metrics.The remaining sections of this blog will create the setup depicted in the above diagram. Airflow metrics stored in Prometheus can then be viewed in the Grafana dashboard. This endpoint is periodically scraped by the Prometheus server, which persists the metrics in its database. The statsd_exporter aggregates the metrics, converts them to the Prometheus format, and exposes them as a Prometheus endpoint. The solid line starting at the Webserver, Scheduler, and Worker shows the metrics flowing from the Webserver, Scheduler, and the Worker to the statsd_exporter. The diagram depicts three Airflow components: Webserver, Scheduler, and the Worker. Overall, the Airflow monitoring diagram looks as follows: The Prometheus server can then scrape the metrics exposed by the statsd_exporter. The statsd_exporter receives statsd metrics on one side and exposes them as Prometheus metrics on the other side. How can the statsd metrics be sent to Prometheus? It turns out that the Prometheus project comes with a statsd_exporter that functions as a bridge between statsd and Prometheus. Our goal though, is to send the metrics to Prometheus. These metrics would normally be received by a statsd server and stored in a backend of choice. Apache Airflow can send metrics using the statsd protocol. This blog covers a proof of concept, which shows how to monitor Apache Airflow using Prometheus and Grafana.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |