By Surajtikoo．Apr 1, 2023
As we know, The Datadog Agent uses the REST API to communicate with the Datadog servers and send data to be processed and analyzed. The REST API is a set of HTTP endpoints that the Agent can send requests to in order to perform various tasks, such as sending metrics, events, and traces, as well as configuring the Agent itself.
When you install the Datadog Agent, you need to configure it with an API key. This key is used to authenticate the Agent with the Datadog servers and ensure that only authorized users can access the data being sent.
It is a common practice if we have an application running on AWS we normally store these credentials in either a secret Manger or parameter store which is the secure way to store these credentials but it becomes challenging when there are multiple teams using the same credentials.
The below depicts how these credentials are getting used by multiple teams
The above architecture diagram displayed above demonstrates that Team A, B, and C are all storing their credentials within their respective AWS accounts. While this approach may seem viable for a small number of teams, it becomes increasingly difficult to manage as more teams are added to the system.
Challenges
- Rotating keys can be challenging, as can sharing new keys across multiple teams.
- Additionally, if keys are compromised, there is a high risk of them being compromised again.
To overcome these obstacles, a centralized approach has been adopted for storing credentials. By centralizing these credentials, the risks associated with managing them are greatly reduced.
A high-level architecture diagram for this approach is included below.
The architecture diagram above illustrates a custom solution that has been developed & hosted on AWS in a shared account which allows us to store these credentials centrally.This solution utilizes several services to achieve its goals.
- The first service used is the CloudWatch Event Rule, which triggers a Lambda function to run at a desired frequency. The frequency can be chosen based on specific needs.
- In the second step, the lambda function calls the Datadog API call to create the new API Key. Below is the Datadog API documentation with more details
Note: To access the API authentication is needed which is the Datadog API key and APP Key. We are storing the datadog API key in the secret Manager.
3. Next, in the 3rd step, the Lambda function stores the newly generated keys in the Secret Manager.
4. Once the Secret is updated, a notification is sent to all teams using the SNS.
5. A Cross role account is created for all teams with the least privileges i.e ready-only access. All the teams will rerun their pipeline via Jenkins which will take the latest secret from the shared account secret manager.
Note: The security of this architecture is maintained by deploying the Lambda function in a VPC. Additionally, SSL certificates are used to secure communications when making Datadog API calls.
Below is the sample code to trigger the Datadog API
Conclusion
Overall, this custom solution provides a secure and efficient way to store and rotate keys centrally, with notifications being sent to all relevant teams to ensure they have access to the latest information.
The original article published on Medium.
