Prometheus + Grafana in Django

Just developing things , testing and deploying to server is not enough to ensure up-time SLA. Once things are deployed in production environment, in-spite of rigorous testing in staging environment, things may go wrong in production.

Some of the reasons for malfunctioning in production are:

• Too many request which were not expected.
• Unexpected infra problem such as disk full.
• Missed test case etc.

No code and human are 100% perfect

But since we have to meet our SLA, for a better user experience. We can reduce the blast radius after the code is deployed in the server, for any fault.

Here comes the concept of Monitoring. Once the code is deployed in production environment, one should continuously monitor their changes in production. Continuously monitoring server manually may also be problematic, as there are chances for miss there.

We can overcome this problem with monitoring tools available in market. Some are open-source and some comes with a paid license.

Some tools we can use for monitoring:

• Zabbix
• Cadvisor
• Prometheus
• Grafana

In this tutorial we’ll be using Prometheus & Grafana for our monitoring needs.

Step 1:

Setting up Prometheus & Grafana with docker-compose.

• Configure Prometheus config (save below file as prometheus.yml)

The above file is a configuration file for Prometheus, below is the explanation.

1st part is the global config which has 2 variable scrape_interval and evaluation_interval, these specify what should be the frequency of the scraping and evaluation of the data that is emitted out of your project.

2nd part is scrape_config, which basically states from where the data is to be scraped. Please take a note of the metrics_path here (will explain its significance later in this tutorial)

Prometheus works on the principle of scraping the data out of an emitter. We need a emitter of data in our django project which will be in the last step of this tutorial (stay tuned).

• Create and run docker-compose.yml file

Here we have used 2 volumes to keep our data persistent one for each instance, also we’ll keep our network mode host in the above file so that, local django server can be accessed within the docker environment.

To run compose file.

docker-compose up -d

Wait for few seconds to boot it up, and open http://localhost:9090/targets in your browser (this is Prometheus GUI). You will see below screen, but note we’ve not yet configured the django emitter, so the status might be DOWN or UNKNOWN.

Now visit http://localhost:3000/ to check whether grafana is up and running, here username and password you can set (initially username and password as “admin” works)

Step 2:

Configure Django emitter:

Add below lines in your settings file, what it does is add Prometheus middleware at start and end, add Prometheus library in installed apps list.

MIDDLEWARE = \
    ['django_prometheus.middleware.PrometheusBeforeMiddleware'] + \
    MIDDLEWARE + \
    ['django_prometheus.middleware.PrometheusAfterMiddleware']

INSTALLED_APPS += ['django_prometheus']

Add url for the emitter.

As said above the significance of metrics_path, here is where it’s actually used. The Prometheus url is publicly exposed as your other endpoints, to add a small layer of security we added -xyzabc suffix here (Note: please do not use such simple suffix).

Finally install the Prometheus client library.

pip3 install django-prometheus==1.0.15

Once this django configuration is done and is up and running, we can see the status change in our Prometheus GUI

Step 3:

Final Step → Create a dashboard to monitor (Note its a basic dashboard, there are many possibilities on how we can create it).

Create new dashboard, save it, go to settings.

On settings panel, we’ll see JSON Model option in right side. Click on that and paste the following keeping the title, uid, version same as it was original.

With this we complete our basic dashboard setup too. Below is the result of our hard work.

But that’s not it. We need to add alerts for worst case scenario. Grafana provides us with various alerting channels such as slack, telegram, webhook etc.

For this tutorial purpose, we’ll check slack notification. To get slack notification we first need a slack token, you can create yours with https://api.slack.com/authentication/basics. Once that’s done follow below path to add a channel.

1. Fill in name of channel
2. Type of notification should be slack
3. Fill in your slack channel name (should exist in your account)
4. Fill in your token

Once that is done click on Test and Save

Now we create an alert for a panel element (here its CPU usage alert)

Once set, we will receive alert for CPU usage avg is above 90, so that we can take necessary action related to this. Similar to this we can have an alert on various monitoring actions we configure.

Thanks for reading this far ✌🏼