By Surajtikoo．May 24, 2023
In today’s world of distributed systems and microservices, managing and monitoring logs across multiple services can be challenging. That’s why having a centralized logging solution is crucial to streamline operations and gain valuable insights. One such solution that has gained popularity is Datadog, which offers a comprehensive monitoring platform for infrastructure, applications, and logs.
When it comes to handling customer data in a microservices environment, data privacy and security play a very vital role. It is essential to ensure that sensitive customer information, such as personal identifiers, email IDs, customer names, or financial data, is not inadvertently exposed to third-party monitoring tools. Following best practices, teams should be cautious about sharing such information to minimize the risk of data leaks and potential legal implications.
The below picture depicts how different microservices are sending the application logs to the centralized monitoring tool.
However, even with the best intentions, mistakes can happen. Human error or oversight may lead to the accidental inclusion of sensitive information in logs, exposing it to unauthorized access.
This often happens when developers enable “DEBUG” logs. To address these risks and enhance data privacy, organizations should take proactive measures.
This is where the recently introduced feature of Datadog, known as “Datadog Scanner”, comes into play. The Datadog Scanner is a powerful tool that automates the process of identifying and redacting sensitive customer information from logs before they are sent to the monitoring platform. By leveraging advanced machine learning algorithms, the scanner intelligently detects patterns and keywords indicative of sensitive data, such as credit card numbers or social security numbers. It then replaces or redacts this information with anonymized placeholder values, ensuring the protection of customer data throughout the monitoring process.
Implementing the Datadog Scanner provides organizations with several key benefits which are as follows
- It significantly reduces the risk of accidental exposure of sensitive customer information, safeguarding both the organization and its customers from potential data breaches.
- It helps organizations comply with data privacy regulations by ensuring that personally identifiable information (PII) is appropriately handled and protected. This not only minimizes legal and reputational risks but also demonstrates a commitment to data privacy.
Moreover, the Datadog Scanner offers customization options, allowing organizations to tailor the scanning process to their specific needs. This flexibility ensures that sensitive information is accurately identified and redacted, aligning with industry regulations and internal policies.
The below picture depicts the various different options that Datadog Data Scanner provides.
From the above architecture diagram, we can enable the data scanner at log, APM, and Rum Agent. For my POC we have performed the scanning for the logs.
The steps to define the Data Scanner in Datadog are as follows
- Define the Datascanner group under the Sensitive Data scanner. This can be easily created via the console but we recommend to create it via the Terraform. Datadog provides the integration for the terraform.
- Under the scanner group create the scanning rule. Datadog provides a couple of predefined scanning rules. As of now, we have used the email address scanner.
3. Enable the scanning. Once the scanning is enabled if there are any logs with the email. The scanner will automatically encode the piece.
Before Data scanner
After the Data scanner, we can see the email part is encoded.
Datadog also provides the default dashboard with all desired metrics. We can easily add the alert based on whenever any event is triggered by the data scanner in the logs
Conclusion
In conclusion, Datadog Data Scanner is a powerful tool that enhances data security by continuously monitoring and detecting potential threats in real-time.
The original article published on Medium.
