A Practical Guide to App Monitoring with Datadog for Python APIs

By Sourav Verma．May 2, 2022

Application monitoring is one of the most important pieces of any system design. It helps us to review and analyze our servers for operations-related tasks like availability, metrics, containers, processes-running, performance, networks, logs, traces, security, etc. Since the application’s health depends largely on the health of the underlying server, it is essential to use some good server monitoring tools, which give us the above-mentioned capabilities.

In this article, I’ll be talking about Datadog installation for Python APIs (Flask + Gunicorn), which I believe is one of the best all-around tools out there.

Note: All the following steps are performed on Ubuntu 18.04.2 LTS System.

---

1. Get started with datadog

The first step would be to create a 14-days trial account on Datadog (Assuming you don’t have an account yet). You can use any region to house your data (I’ll be using eu as the region) and fill-out other details. You’re signed up now.

You’ll get a login credential and an API_KEY, which we’ll be using throughout.

Now, You can monitor as many servers as you like for this period and get a taste of what Datadog is capable of 👍

2. Datadog Agent Installation

To collect metrics, request traces, and logs from your application, you’ll need to install the latest version of the Datadog Agent on your application host. You can download and install the Agent with the following command, which will prompt you for your system root password:

DD_API_KEY=<API_KEY> DD_AGENT_MAJOR_VERSION=7 DD_SITEbash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

3. Configure the Datadog Agent

a) Open up the datadog.yaml configuration file in the Agent directory:

vi /etc/datadog-agent/datadog.yaml

To enable log collection, set log_enabled: true in /etc/datadog-agent/datadog.yaml

Also to trace processes, set

process_config:
    enabled: 'true'

in /etc/datadog-agent/datadog.yaml

b) Restart the Agent to reload the configuration file: service datadog-agent restart

4. Collecting App logs

a) mkdir conf.d/python.d/ #create a python directory under conf.d

b) vi conf.d/python.d/conf.yaml #create and edit the config file

c) Add the following lines to the conf.yaml file (Indentation is Important)

init_config:

instances:

##Log section
logs:

  - type: file
    path: "<PATH_TO_PYTHON_LOG>.log"
    service: "<YOUR_APPLICATION>"
    source: python
    sourcecategory: sourcecode
    # For multiline logs, if they start by the date with the format yyyy-mm-dd uncomment the following processing rule
    #log_processing_rules:
    #  - type: multi_line
    #    name: new_log_start_with_date
    #    pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])  # If multiple log files to be logged: repeat the above line of codes with different path and service as needed

d) Restart the datadog agent: service datadog-agent restart

5. Setting up the app performance monitoring

a) source activate <conda_env>; pip install ddtrace

b) Update the startup script: ddtrace-run gunicorn main:app [...]

c) Restart the application

6. monitoring Gunicorn metrics (optional)

a) source activate <conda_env>; pip install setproctitle

b) create a conf.yaml file to start collecting gunicorn metrics

sudo cp /etc/datadog-agent/conf.d/gunicorn.d/conf.yaml.example /etc/datadog-agent/conf.d/gunicorn.d/conf.yaml

c) Add the following to your newly created conf.yaml file, where my_web_app is the name of your application (you can check your application’s name using the command ps -ef | grep 'gunicorn'):

init_config:

instances:
    ## @param proc_name - string - required
    ## The name of the gunicorn process. For the following gunicorn 
    ##
    ## gunicorn --name <WEB_APP_NAME> <WEB_APP_CONFIG>.ini
    ##
    ## the name is `<WEB_APP_NAME>`
  - proc_name: my_web_app

d) Update the startup script: ddtrace-run gunicorn --name my_web_app --statsd-host=localhost:8125 main:app [...]

e) Configure the app to DogStatsD:

• source activate <conda_env>; pip install datadog
• code the following in your main API file:

from datadog import initialize, statsd

options = {
    'statsd_host':'127.0.0.1',
    'statsd_port':8125
}

initialize(**options)

f) Restart the datadog agent: service datadog-agent restart

g) Restart the gunicorn agent: service gunicorn restart

h) To validate if it is configured properly:

$ sudo netstat -nup | grep "127.0.0.1:8125.*ESTABLISHED"
udp        0      0 127.0.0.1:38374         127.0.0.1:8125          ESTABLISHED 15500/gunicorn: mas

i) Restart the application

7. To check the status of Datadog:

Running datadog-agent statusshould give output similar to the followings:

[...]

  gunicorn
  --------
  Total Runs: 1
  Metrics: 2, Total Metrics: 2
  Events: 0, Total Events: 0
  Service Checks: 1, Total Service Checks: 1

[...]

=========
DogStatsD
=========

Checks Metric Sample: 882
Event: 1
Events Flushed: 1
Number Of Flushes: 4
Series Flushed: 403
Service Check: 64
Service Checks Flushed: 52

That’s it!

You’ve now successfully configured Datadog for your API, go to the dashboard and monitor all the metrics. 💁

Conclusion

Kudos to you for reaching this far. I hope you found this article useful.

Hopefully, I’ve covered the Datadog for Python well enough in this article to give you a basic understanding of this wonderful tool and help you get started with the same.

Please share your views in the comment section or any suggestions for further articles. Cheers!

---

Sourav Verma

• If you enjoyed this, follow me on medium for more such Articles.
• Connect with me on LinkedIn.

---

The original article published on Medium.