Prometheus and Grafana

Introduction

In the world of software development, understanding how your applications perform in real-time is crucial. Observability tools help you monitor and visualize metrics, logs, and traces to gain insights into your application's behavior. Two popular open-source tools for observability are Prometheus and Grafana. Prometheus is a powerful monitoring system that collects metrics from various sources, while Grafana provides an intuitive interface for visualizing these metrics.

In this tutorial, we'll explore how to set up Prometheus for monitoring and Grafana for visualization. We'll cover the basics of both tools and provide practical examples to help you get started.

Concept

Prometheus

Prometheus is a time-series database that collects metrics from targets at specified intervals. It uses a pull model, where it scrapes metrics from instrumented applications or exporters. Prometheus supports multi-dimensional data models with time series data identified by metric name and key/value pairs called labels.

Key Features of Prometheus:

• Scalability: Designed to handle large volumes of metrics.
• Flexibility: Supports various types of targets (e.g., HTTP, Pushgateway).
• Alerting: Can trigger alerts based on the collected metrics.
• PromQL: A powerful query language for time series data.

Grafana

Grafana is a visualization tool that works with Prometheus and other data sources to create dashboards. It allows you to visualize your data in various ways, such as graphs, tables, and singlestat panels.

Key Features of Grafana:

• Visualization: Supports multiple types of visualizations.
• Dashboards: Can be customized and shared across teams.
• Plugins: Extensible with a wide range of plugins for additional functionality.
• Alerting: Integrates with Prometheus for alert notifications.

Examples

Setting Up Prometheus

1. Install Prometheus: You can download Prometheus from the official website. For this example, we'll use version 2.34.0.

   wget https://github.com/prometheus/prometheus/releases/download/v2.34.0/prometheus-2.34.0.linux-amd64.tar.gz
   tar xvfz prometheus-2.34.0.linux-amd64.tar.gz
   cd prometheus-2.34.0.linux-amd64
   

2. Configure Prometheus: Edit the prometheus.yml file to specify the targets you want to monitor.

   global:
     scrape_interval: 15s
   
   scrape_configs:
     - job_name: 'prometheus'
       static_configs:
         - targets: ['localhost:9090']
   

3. Start Prometheus: Run Prometheus using the configuration file.

   Terminal

   ./prometheus --config.file=prometheus.yml

   You can access the Prometheus UI at http://localhost:9090.

Setting Up Grafana

1. Install Grafana: Download and install Grafana from the official website.

   wget https://dl.grafana.com/oss/release/grafana-8.3.4.linux-amd64.tar.gz
   tar xvfz grafana-8.3.4.linux-amd64.tar.gz
   cd grafana-8.3.4/bin
   

2. Start Grafana: Run Grafana using the default configuration.

   Terminal

   ./grafana-server

   You can access the Grafana UI at http://localhost:3000. The default login credentials are admin/admin.

3. Add Prometheus as a Data Source:

   • Log in to Grafana.
   • Navigate to Configuration > Data Sources.
   • Click on "Add data source".
   • Select "Prometheus" and enter the URL of your Prometheus server (e.g., http://localhost:9090).
   • Save and test the connection.

4. Create a Dashboard:

   • Go to Create > Dashboard.

   • Add a new panel.

   • Use PromQL to query metrics from Prometheus. For example, you can use:

     rate(http_requests_total[1m])
     

   • Configure the visualization and save the dashboard.

Practical Example

Let's create a simple example where we monitor the CPU usage of a server using Prometheus and visualize it in Grafana.

1. Install Node Exporter: Node Exporter is a Prometheus exporter for hardware and OS metrics. Install it on your server.

   wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
   tar xvfz node_exporter-1.2.2.linux-amd64.tar.gz
   cd node_exporter-1.2.2.linux-amd64
   

2. Start Node Exporter: Run Node Exporter.

   Terminal

   ./node_exporter

3. Configure Prometheus to Scrape Node Exporter: Update the prometheus.yml file to include Node Exporter as a target.

   scrape_configs:
     - job_name: 'prometheus'
       static_configs:
         - targets: ['localhost:9090']
     - job_name: 'node'
       static_configs:
         - targets: ['localhost:9100']
   

4. Restart Prometheus: Restart Prometheus to apply the changes.

   Terminal

   ./prometheus --config.file=prometheus.yml

5. Create a Grafana Dashboard for CPU Usage:

   • In Grafana, create a new dashboard.

   • Add a panel and use the following PromQL query to visualize CPU usage:

     rate(node_cpu_seconds_total{mode="user"}[1m])