The second hypergrowth stock that billionaires couldn&#x27;t stop buying in the March-ended quarter as they were sending shares of Nvidia to the chopping block is cloud-based observability company <strong>Datadog</strong> (DDOG -0.35%). Similar to Okta, five billionaires took the plunge in the first quarter, including ...

Forget Nvidia: Billionaires Are Selling It and Buying These 2 Hypergrowth Stocks Instead @themotleyfool #stocks $OKTA $NVDA $DDOG

Get the most recent info and news about Datadog on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Datadog, Inc. provides monitoring and analytics platform for developers, information technology operations teams, and business users in the cloud in North America and internationally. The company's SaaS platform integrates and automates infrastructure monitoring, application performance monitoring, log management, and security monitoring to provide real-time observability of its customers technology stack. Its platform also provides user experience monitoring, network performance monitoring, cloud security, developer-focused observability, and incident management, as well as a range of shared features, such as dashboards, analytics, collaboration tools, and alerting capabilities. The company was incorporated in 2010 and is headquartered in New York, New York.

Datadog

Monitoring Monolith With Datadog: How to Avoid the Bystander Effect

ডেটাডগের সাথে মনোলিথ পর্যবেক্ষণ: কীভাবে বাইস্ট্যান্ডার প্রভাব এড়ানো যায়

Monolith überwachen mit Datadog: So vermeiden Sie den Bystander-Effekt

Monitoreo de monolito con Datadog: cómo evitar el efecto espectador

Surveillance de Monolith avec Datadog : comment éviter l'effet spectateur

डेटाडॉग के साथ मोनोलिथ की निगरानी: बाईस्टैंडर प्रभाव से कैसे बचें

Datadog によるモノリスの監視: 傍観者効果を回避する方法

Datadog으로 모놀리스 모니터링: 방관자 효과를 피하는 방법

Monitorando Monolith com Datadog: como evitar o efeito espectador

Мониторинг монолита с помощью Datadog: как избежать эффекта свидетеля

Monolith'i Datadog ile İzleme: Seyirci Etkisinden Nasıl Kaçınılır

Giám sát Monolith bằng Datadog: Cách tránh hiệu ứng người ngoài cuộc

使用 Datadog 监控整体式应用：如何避免旁观者效应

If you are using Databricks serving endpoint, and you wish to export metrics to Datadog, you can face with some challenges in Datadog documentation. This article will help you to solve them, at least it will help you to make a PoC. Here are list of links which you can use: https://docs.databricks.com/en/machine-learning/model-serving/metrics-export-serving-endpoint.html https://docs.datadoghq.com/agent/basic_agent_usage/ubuntu/?tab=agentv6v7 Databricks serving endpoint is a service which allows you to deploy your model and serve it. It is a part of Databricks MLflow. You can find more information about it here: https://docs.databricks.com/applications/mlflow/model-serving.html Once you created an endpoint, out of the box you can see some metrics in Databricks UI. Also list of metrics can be received via REST API: curl --request GET "https://${DATABRICKS_HOST}/api/2.0/serving-endpoints/{endpoint_name}/metrics" --header "Authorization: Bearer ${DATABRICKS_TOKEN}" But if you want to export them to Datadog, you need to install and configure Datadog agent. Pre-requisites Created Databricks serving endpoint Bearer PAT token behalf of Databricks user/service principal, which has `CAN_QUERY` permission on the endpoint Datadog API key Enabled OpenMetrics integration in your Datadog account, link: https://app.datadoghq.com/integrations Install Datadog agent Datadog offers several ways to install agent, they are listed here: , however looks like some of them are not working, or at least I was not able to make them work. That is why I decided to use the most simple way — install agent on Docker. https://app.datadoghq.com/account/settings/agent/latest?platform=overview Before agent installation, you need to create a file `datadog.yaml`, which should be located ` `, with the following content: /etc/datadog-agent/datadog.yaml api_key: "__DD_API_KEY__"
site: "datadoghq.com"
hostname: "__YOUR_HOST_NAME__" Then you need to create `metric-config.yaml` file, which should be located ` `, with the following content: /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default instances:
 - openmetrics_endpoint: https://__DATABRICKS_HOST__/api/2.0/serving-endpoints/__ENDPOINT_NAME__/metrics

   metrics:
     - cpu_usage_percentage:
         name: cpu_usage_percentage
         type: gauge
     - mem_usage_percentage:
        name: mem_usage_percentage
        type: gauge
     - provisioned_concurrent_requests_total:
        name: provisioned_concurrent_requests_total
        type: gauge
     - request_4xx_count_total:
        name: request_4xx_count_total
        type: gauge
     - request_5xx_count_total:
        name: request_5xx_count_total
        type: gauge
     - request_count_total:
        name: request_count_total
        type: gauge
     - request_latency_ms:
        name: request_latency_ms
        type: histogram

   tag_by_endpoint: false

   send_distribution_buckets: true

   headers:
     Authorization: Bearer __DATABRICKS_TOKEN__
     Content-Type: application/openmetrics-text The full possible options for ` ` can be found in template file which is located in a container under ` ` conf.yaml.default /opt/datadog-agent/agent/conf.d/openmetrics.d/ I’ve created the next , because I have some difficulties with that one which is provided in Datadog documentation: Dockerfile FROM ubuntu:20.04

# Set environment variables
ENV DD_API_KEY=[SPECIFY_DD_API_KEY]
ENV DD_SITE=datadoghq.com
ENV DD_HOSTNAME=datadog-agent-ubuntu
ENV DATABRICKS_HOST=[SPECIFY_DATABRICKS_HOST]
ENV DATABRICKS_TOKEN=[SPECIFY_DATABRICKS_TOKEN]
ENV ENDPOINT_NAME=[SPECIFY_ENDPOINT_NAME]

# Install curl
RUN apt-get update && apt-get install -y curl

# Copy the datadog.yaml file into the Docker image
COPY datadog.yaml /etc/datadog-agent/datadog.yaml
RUN sed -i -e "s@__DD_API_KEY__@$DD_API_KEY@g" /etc/datadog-agent/datadog.yaml

# Install the Datadog agent
RUN bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

# Copy the metrics-config.yaml file into the Docker image
COPY metrics-config.yaml /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN sed -i -e "s@__DATABRICKS_HOST__@$DATABRICKS_HOST@g; s@__ENDPOINT_NAME__@$ENDPOINT_NAME@g; s@__DATABRICKS_TOKEN__@$DATABRICKS_TOKEN@g; " /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN service datadog-agent restart

# Run a process that doesn't exit
CMD ["tail", "-f", "/dev/null"] Usage Run serving endpoint Define environment variables in `Dockerfile’ Build docker image Run docker container # Step 1: Build the Docker image
# Navigate to the directory containing the Dockerfile
cd /path/to/directory/with/dockerfile

# Build the Docker image
# Replace 'my-image' with the name you want to give to your Docker image
docker build -t datadoc-agent-image .

# Step 2: Run a container from the built image
# Replace 'my-container' with the name you want to give to your Docker container
docker run --name datadog-agent -d datadoc-agent-image

# Step 3: Connect to a container and restart agent
docker exec -it datadog-agent  bash
service datadog-agent restart Some usefully commands for DataDog agent: datadog-agent status
datadog-agent health

service datadog-agent start
service datadog-agent restart Also appears . here

How to Export Metrics from Databricks Serving Endpoint to Datadog

Effective metrics and monitoring play a key role in high-quality development, bug fixes, and handling of user requests and incidents. At Social Discovery Group, we employ a diverse range of tools to evaluate the performance of our products. In this article, we will share some hacks of how we managed to build a stable application monitoring and metrics system and reduced bug detection time on our products with the help of Datadog. SDG products are helping more than 250 mln people connect and build relationships globally, and our user base is constantly growing. The main factor in this success is our ability to promptly respond to user needs. We have a lot of experience working with various monitoring systems, including Datadog. Here's why: Currently, Datadog offers a variety of integration options with systems at different levels. You can explore these capabilities . here Datadog provides comprehensive documentation for configuring integrations with various services. We managed to build strong partnership relations. Here, we're sharing our experience in setting up and configuring APM metrics for applications within our Kubernetes cluster for Datadog. The article won't cover the deployment of projects in AKS, the CI/CD process, and other DevOps details. Instead, we will focus on the finer points of configuring Datadog monitoring for APM metrics. The technology stack used: Azure Services, Azure Kubernetes Service (AKS), ASP .Net Core 7, Datadog. For monitoring applications and services, we utilize the Datadog agent deployed within the cluster through a Helm chart with additional parameters from the values.yaml file. In the main agent configuration file, you'll need to enable the DogStatsD module and specify the port (default is 8125). For the data sent from an external host, the Datadog agent requires the following configuration: dogstatsd_non_local_traffic: true and apm_non_local_traffic: true. Here is the values.yaml file for one of the clusters, with some variables being passed at the deployment stage. The deployment is produced with Azure Devops. " 
datadog:
  apiKey: #{apiKey}#
  appKey: #{appKey}#
  clusterName:  #{ClusterName}#
  kubeStateMetricsEnabled: true
  clusterChecks:
    enabled: true
  dogstatsd:
    useSocketVolume: false
    nonLocalTraffic: true
  collectEvents: false
  apm:
    portEnabled: true #important to include
  env:
    - name: "DD_KUBELET_TLS_VERIFY"
      value: "false"
  systemProbe:
    collectDNSStats: false
  orchestratorExplorer:
    enabled: false
clusterAgent:
  image:
    repository: public.ecr.aws/datadog/cluster-agent
    tag: #{tag}#
  admissionController:
    enabled: false
agents:
  image:
    repository: public.ecr.aws/datadog/agent
    tag: #{tag}#
    doNotCheckTag:  true
clusterChecksRunner:
  image:
    repository: public.ecr.aws/datadog/agent
    tag: #{tag}#
" Then, you need to specify the agent's address for transmitting metrics to Datadog in the application settings. The dashboards for app monitoring are based on the internally used metrics; they had to be created independently. To set up monitoring of ASP.Net services, we used the official documentation that can be found by the . link Since the agent was already configured, one of the methods was to add the necessary lines to the image build and pass variables in the CI/CD system: DD_ENV, DD_SERVICE, DD_AGENT_HOST, to specify the environment, service name, and agent address, respectively. We also need to add the following to the dockerfiles for the services: "
RUN TRACER_VERSION=$(curl -s \https://api.github.com/repos/DataDog/dd-trace-dotnet/releases/latest | grep tag_name | cut -d '"' -f 4 | cut -c2-) \
    && curl -Lo /tmp/datadog-dotnet-apm.deb https://github.com/DataDog/dd-trace-dotnet/releases/download/v${TRACER_VERSION}/datadog-dotnet-apm_${TRACER_VERSION}_amd64.deb

# Copy the tracer from build target
COPY --from=build /tmp/datadog-dotnet-apm.deb /tmp/datadog-dotnet-apm.deb
# Install the tracer
RUN mkdir -p /opt/datadog \
    && mkdir -p /var/log/datadog \
    && dpkg -i /tmp/datadog-dotnet-apm.deb \
    && rm /tmp/datadog-dotnet-apm.deb
 
# Enable the tracer
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
ENV CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
ENV DD_DOTNET_TRACER_HOME=/opt/datadog
ENV DD_INTEGRATIONS=/opt/datadog/integrations.json

" This method works, but it didn't seem like the most optimal solution. We decided to take it a step further and added the following to the deployments of our services: "
metadata.labels:
    tags.datadoghq.com/env: feature
    tags.datadoghq.com/service: service_name
    tags.datadoghq.com/version: '1552'

spec.template.metadata.labels:
    admission.datadoghq.com/config.mode: service
    admission.datadoghq.com/enabled: 'true'
    tags.datadoghq.com/env: feature
    tags.datadoghq.com/service: service_name
    tags.datadoghq.com/version: '1111'

spec.template.metadata.annotations:
    admission.datadoghq.com/dotnet-lib.version: v2.38.0

spec.template.spec.containers.name.env:
    - name: DD_TRACE_AGENT_URL
      value: datadog-agent.monitoring
    - name: DD_TRACE_STARTUP_LOGS
      value: 'true'
    - name: DD_LOGS_INJECTION
      value: 'true'
    - name: DD_RUNTIME_METRICS_ENABLED
      value: 'true'
    - name: DD_PROFILING_ENABLED
      value: 'true'
    - name: DD_APPSEC_ENABLED
      value: 'true'
" That’s what was changed in the agents: "
datadog:
  apm:
    socketEnabled: true
    portEnabled: true
    enabled: true
clusterAgent:
  admissionController:
    enabled: true
    mutateUnlabelled: false
providers:
  aks:
    enabled: true
" After all these steps, the data with highly detailed metrics for all services started flowing in Datadog under the APM -> Services section, and the graphs were automatically displayed. We had to tinker with the annotations settings for the second method; not everything started working smoothly right away. Regarding the notification system, it's worth mentioning that it is user-friendly and intuitive in Datadog. Notifications are created in the "Monitors -> Manage Monitors" section. The improvements we described above yielded several valuable outcomes. We now have a deeper understanding of how our system operates and adapts to various changes. Additionally, we've established a stable application monitoring and metrics system that operates independently of service builds, helping to reduce bug detection times. This, in turn, has allowed us to optimize our services and improve development speed and overall system quality.

How to Use Datadog for the APM Metrics Application

এপিএম মেট্রিক্স অ্যাপ্লিকেশনের জন্য ডেটাডগ কীভাবে ব্যবহার করবেন

So verwenden Sie Datadog für die APM Metrics-Anwendung

Cómo utilizar Datadog para la aplicación de métricas APM

Comment utiliser Datadog pour l'application APM Metrics

एपीएम मेट्रिक्स एप्लिकेशन के लिए डेटाडॉग का उपयोग कैसे करें

APM メトリクス アプリケーションに Datadog を使用する方法

APM 지표 애플리케이션에 Datadog을 사용하는 방법

Como usar o Datadog para o aplicativo de métricas APM

Как использовать Datadog для приложения APM Metrics

APM Metrics Uygulaması için Datadog Nasıl Kullanılır?

Cách sử dụng Datadog cho ứng dụng Số liệu APM

如何使用 Datadog 进行 APM 指标应用程序

At , we currently use as our deployment platform, a “roll your own Platform-as-a-Service” that you can install to handle container orchestration on AWS’s ECS (Elastic Container Service). Lob Convox Convox is showing its age, and this year we began the process of replacing Convox with HashiCorp’s , a flexible workload orchestrator to deploy and manage our containers on AWS. Nomad As we prepared our migration to Nomad, our biggest concern was performance. We started our tests with a barebones proof-of-concept setup of our API in Nomad. We benchmarked and compared the two systems and found Nomad was running slower. This was extremely surprising as Convox (which leverages AWS Elastic Container Services) and Nomad both use Docker containers under the covers. We assumed going from older to newer technology would result in some performance gains. Why Is Our Performance Suffering? Due to the stripped-down Nomad cluster, we were initially not set up with Datadog’s agent to collect application traces and profiles. Our initial performance testing revealed that every request performed poorly with a four-second delay regardless of load, which was unexpected. Even stranger, running a load test showed performance got better and would go up and down over time. Internally, we debated whether poor database performance was the cause. However, with the time-honored process of deleting code and observing the results, we found that the four-second delay occurred on save requests to our AWS S3 bucket. To determine if the problem was in our code or the infrastructure we needed to isolate the suspect code and configure Datadog to effectively trace interactions between our app and the infrastructure. We did this by extracting the S3 upload method from our production codebase to eliminate noise in our trace, configuring the Datadog APM trace library for this code, and enabling the Datadog agent on our Nomad cluster. We found 4 sequential calls with exclamation marks in Datadog. Why would an S3 upload cause 4 HTTP PUT requests? And why did each request fail after exactly 1 second? Looking closer, we found socket hang-ups and an IP address being called. The AWS JavaScript SDK makes 4 attempts to call the Elastic Compute Cloud (EC2) metadata service (that’s the 169.254.169.254 IP address), then gives up and tries something else before successfully uploading the file to S3. This exonerated our developers as it clearly wasn’t a problem with our code. Something wasn’t right with our infrastructure. Let’s zoom out for a minute and discuss how Amazon services share information with each other.  All AWS API requests have to be signed with credentials that tie back to an IAM Role with specific permissions, like uploading objects to our AWS S3 bucket for example. Traditionally, this role would be associated with access keys and those credentials would be loaded into your application -- . potentially exposing them AWS provides a more secure mechanism when the calls originate from an AWS service. For our existing Convox services, this was ECS Container Instance Roles. For services running on ECS with an associated instance role, the AWS SDK would automatically call the metadata API on our ECS cluster to get temporary credentials it can use to access other Amazon services like S3. This means we don’t need to include our AWS credentials in our code and the IAM permissions can be directly tied to the ECS service itself. For EC2, there exist EC2 Instance Profiles, which function almost the same. Almost. Why Can’t, Can’t, Can’t, Can’t, and Finally Can the AWS Sdk Reach the Metadata API on My EC2 Instance? So, why are the calls to the EC2 metadata service failing, yet somehow still succeeding in the end?  After hours of tearing through all the JavaScript AWS SDK documentation and finding nothing, we turned to Google. Searching for the IP address we’d seen in Datadog led to documentation on the EC2 instance metadata API: The AWS SDKs use IMDSv2 calls by default. If the IMDSv2 call receives no response, the SDK retries the call and, if still unsuccessful, uses IMDSv1. This can result in a delay. In a container environment, if the hop limit is 1, the IMDSv2 response does not return because going to the container is considered an additional network hop. To avoid the process of falling back to IMDSv1 and the resultant delay, in a container environment we recommend that you set the hop limit to 2. For more information, see . Configure the instance metadata options Translation: In order for the SDK, running in a Nomad-orchestrated container, to obtain credentials it must pass through the EC2 instance to reach EC2 metadata API. This counts as two “hops” and the default hop limit is one! We changed the hop limit to two and the request time dropped from 4 seconds to 85ms. Here is the Datadog APM trace after increasing the hop limit to two: What’s Wrong With This Picture? To put it bluntly, EC2 instance profiles are not the proper way to provide credentials to applications running in a container orchestrator. They are too coarse-grained. Permissions are granted to applications running on the EC2 instance, not a specific application or service. all After all, that’s why ECS, which runs on EC2 instances and therefore could use the EC2 instance profiles, instead relies on ECS instance profiles. So why did we? Because we needed a temporary solution in order to trial our proof-of-concept Nomad cluster, and EC2 instance profiles were tantalizingly similar to ECS instance profiles -- our bread-and-butter solution in Convox. Oh, and they work. The application was able to access the private S3 bucket. Well, eventually. Mostly. do The problem is the AWS JavaScript SDK call eventually succeeds which masks the underlying issue. This violates the . Basically, a system should behave in a way that most users will expect it to behave. principle of least surprise The behavior should not astonish or surprise users. You could describe the SDK behavior as a pseudo-silent failure. Our application performance was severely degraded and we were unaware anything had gone wrong. As developers, we expect things to succeed or fail, not mostly fail and then succeed. You might be wondering why isn’t AWS setting the hop limit default higher. We don’t know for certain, but our guess would be for security purposes. A low default hop limit ensures that credential requests originate inside the EC2 instance and forces developers to make a choice to increase the hop limit and be aware of any associated risks. Conclusion Several lessons can be drawn from our experience tracking down this performance issue. Regularly test the performance of your application Use modern tools like Datadog when investigating performance issues Isolate suspect code to remove the noise and observe the signal Dig into the documentation Google can help your investigation when you uncover unique clues (but you already knew that)

How Datadog Revealed Hidden AWS Performance Problems

Some time back, our engineering leaders suddenly noticed dramatically increased billings from AWS that far outpaced typical benchmarks. The problem turned out to be data insights tools sending an unexpectedly large volume of data. The solution required a deep dive into how AWS billing works and an intricate series of steps to discover the location of the firehose of data. Developer Advocate, Aydrian Howard, sat down with CTO Seth Carney to review and discuss the experience and how the team ultimately managed to control and reduce AWS costs. Courier’s Aydrian: All right. Hello, Seth. Seth: Hey, Aydrian. How's it going? How AWS credits from the YC program made life easier Aydrian: Good. So Courier is coming up on two years old. We're a YC company, which means we have some benefits that come along with that. Can we start by talking about the AWS benefits? Seth: Yeah, absolutely. So one of the really fantastic benefits of going through the YC program is that there is a fairly sizable chunk of credits that you can apply to your AWS account. With such a great benefit, we've been able to focus on investing the capital we would have spent there on other areas of Courier. Aydrian: Nice. So, this is really a benefit that not a lot of people get. How does this affect the way that we built Courier? If we did not have these credits, what would we have done differently from your perspective? Seth: Interesting question. Courier has got a floor as far as what we minimally have to pay to keep that infrastructure running. This includes provisioned concurrency and minimal costs for our Kinesis streams. We didn’t have to spend a lot of time thinking through how we would optimize our per message costs. As an early-stage startup, we have to make strategic decisions around where to invest. Instead of spending thousands of dollars monthly on hosting costs, we were able to focus on growing the team instead. So, I think the biggest thing that it has enabled us to do is really focus on building a product that our customers can find value in. What are our real costs? Aydrian: So like I said, we're coming up on two years. The credits aren't going to last forever. I know that we have started looking into the costs, but really what kicked off this analysis of the cost - what we were paying for AWS? Seth: Yeah. It was an interesting question that our CEO [Troy] posed to me: what was our per message cost? This is a slightly more difficult question to answer than it seems on the surface. To be able to answer this question you have to understand what our floor cost is and what kind of volume that floor supports. And we didn't have that information. We couldn't look at our various sources and say, this is what it costs us to send each and every message that comes out of our platform.. When you're on credits, this is a little harder to figure out than you would think. You start to go back and look at historical costs and all you see is zero, because you didn't pay anything. Aydrian: Exactly. So in looking into this, I guess, how did you go about figuring out what our cost was? And what surprised you about that? Seth: We cast a broad net at first, just trying to look at where we might be able to get some of the costs and information. We obviously started looking to AWS billing utilities for cost data. You could see them in real time by month, but some of the more historical stuff was the challenge. We had a product called Datadog in place that did some cost estimations for us, but they weren’t particularly accurate as far as how some of our configuration is set up. For example, we have a pretty high provision concurrency rate set up on our authorizers since that's a stove-pipe in our application. All of our public facing Lambda functions use a custom authorizer. We wanted to make sure we've got enough capacity to handle normal traffic in addition to having auto-scaling in place to handle spikes in usage. Datadog cost estimation doesn't really take things like that into account. We ended up connecting with a company called CloudZero that thought they could help us gain some of this insight. We integrated their product and started using it to analyze some of the costs per service, different breakdowns like per developer accounts, per service, per tag, and some other various things to start understanding where some of our cost breakdowns were. We definitely found some interesting things, and definitely unexpected things as well. One of the things that stood out immediately was how high our CloudWatch and X-ray spend was. Our CloudWatch spend should have been around one-fifth to one-sixth of our total monthly Lambda invocation costs. Ours was approximately 1.5x our Lambda invocation costs. X-ray was also wildly out of bounds. It was even higher than our Cloudwatch spend. Somewhere in the neighborhood of 3x our Lambda costs. Identifying the problem Aydrian: Let's talk about the steps that we took to identify and resolve the problem. So in using CloudZero, you said it gave you some resolutions. Is that how you went about it? Like we got the analysis and some steps to resolve issues? Seth: The product didn’t necessarily give us the solution. It gave us threads to pull on. We had to do the underlying analysis as to what was causing the increase in spend. The CloudZero team was actually super helpful as well. Ultimately, we had to dig in and figure out what was causing the costs for these services to be outside of expected bounds: determine which of our Lambda functions are generating large volumes of operational data and why.  Did we have log processes in place that should be removed or cleaned up? Was our logging framework configured at the right level? Aydrian: And how did you figure out which functions were generating the most logs or how did you zero in on these different Lambda functions to clean them up? I know we are a serverless company and we probably have lots of Lambda functions that I don't even know about. So are we talking about hundreds of Lambda functions? Seth: That's a good question. I’d say we have around 100 functions. We started by looking at cost breakdowns by cost category to see where the outliers were. We started to tackle Cloudwatch logs first so we started analyzing those by function and log group. We found that there were a couple of things happening. First, we had some cleanup to do. There were a bunch of debug messages that were being output in production. In development in staging you don’t think much of it. But, when you have a function being invoked millions of times. It can add up. In our case, we have functions that are executed more than a hundred million times per month. This had some nominal impact on the log costs, but something was still generating large volumes of log data. It turns out that the Datadog layer that we had put in a few months ago to help get further insights into some of our Lambda functions was injecting the extra data into CloudWatch. So we started this slow process of turning configuration bits off. Unfortunately, no matter how many bits we turned off, there were still minimal amounts of data being pumped in. But, with high requests volumes the extra bits were really adding up. As a test, we removed the layer entirely and saw an instantaneous reduction in both our X-ray costs and our CloudWatch costs. You haven't heard me talk much about X-ray, but I will tell you that there was a constant thread under the current CloudWatch that I was constantly looking at in X-ray where we weren’t making much progress. It turns out the Datadog layer was pumping huge amounts of data into X-Ray even though it was turned off via configuration. The impact of removing the Datadog layer Aydrian: So once you removed the Datadog layer and cleaned up, did usage drop back down to a level that you would expect, or was there more to be done? Seth: Things were back down into nominal zones at that point. Absolutely. Aydrian: Are there insights that we wish we had that we no longer do because we removed this? I know that it's not worth the cost, but are there these insights that it would have been nice to continue to monitor? Seth: I think there are insights that would be nice to have, but not at the expense that they were generating. We would love more accuracy around timeouts of functions and things like that, but we can get in other ways. AWS has also recently released an alternative to the Datadog layer called AWS Lambda insights. We've toyed with replacing it with that, though we haven't yet gone that route. Advice for other companies to reduce AWS costs Aydrian: So for the companies out there who are starting with the serverless model, such as ours, what types of insights for Lambda functions and for serverless architecture would you recommend that people keep an eye on? Seth: There are the pretty obvious things you want to monitor: errors, invocation time, memory utilization, and timeouts. But you also want to make sure you’re monitoring things like cold starts and how much of your provisioned capacity that you are utilizing. In conjunction with autoscaling, this will help you keep latency low for your consumers and scale out when you have unexpected influxes in traffic. If your Lambda functions are connected to SQS Queue or Kinesis Streams or Dynamo Streams, you’ll also want to monitor iterator ages, just to make sure that you're keeping up with the volume that's in the queue. Build up in iterator age might mean you need to adjust your parallelization factor or batch sizes to accommodate increased traffic. Aydrian: So to summarize, we did have a problem with costs. We seem to have those under control. We are looking to add some of the insights back without adding a lot of the costs. Do you have any final tips or recommendations for engineers and startups who are looking to start using serverless and setting this up from scratch? Seth: Both Serverless and AWS have helped us focus on the product we’re trying to build at Courier. But, you have to make sure you understand your ecosystem and use cases. Serverless makes it very easy to opt-in to new and (sometimes) useful features. But, it’s not always particularly easy to understand the cost impact of opting into those services. Especially, as you start to thread multiple services together at the scale of even tens of thousands of requests. I suppose if you're on credits, you can be a little careless (for now). Perhaps more incentive to apply to the YC program! We would love to hear more about your own experiences and issues with sky-high AWS costs. How did you solve the problem? Let us know by tweeting at us @trycourier or by joining our Discord server . discord.gg/courier Author: Seth Carney and Aydrian Howard > Courier employees Also published on: https://www.courier.com/blog/we-changed-datadog-to-avoid-aws-bill

How We Kept Datadog From Blowing Up Our AWS Bill

Getting Logging Right: Observability Foundation

সঠিক লগিং করা: অবজারবেবিলিটি ফাউন্ডেশন

So funktioniert das Logging: Observability Foundation

Lograr el registro correcto: Observability Foundation

Bien gérer la journalisation : Fondation d'observabilité

लॉगिंग को सही तरीके से करना: ऑब्जर्वेबिलिटी फाउंडेशन

ログ記録を正しく行う: Observability Foundation

올바른 로깅 얻기: 관찰 가능성의 기초

Obtendo o registro certo: base de observabilidade

Правильное ведение журналов: Observability Foundation

Günlük Kaydını Doğru Yapmak: Gözlemlenebilirlik Vakfı

Bắt quyền ghi nhật ký: Tổ chức khả năng quan sát

正确记录日志：可观察性基础

In the era of rapid cloud technology advancement, where Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) are integral to numerous projects, the demand for seamless integration of diverse cloud services is undeniable. Microsoft, a major player in this landscape, invested considerable time in crafting the user-friendly Microsoft Entra ID service. However, despite meticulously laid-out instructions from vendors, specialists often encounter issues when configuring integrations with platforms such as Slack, Salesforce, or Datadog. Documentation falls short in providing answers to specific questions. The Social Discovery Group team faced these challenges as well, and in this article, we will offer insights on how to navigate and resolve them. In its portfolio of 50+ dating platforms, the Social Discovery Group utilizes various technologies and cloud services to guarantee their security and reliability. We have extensive experience in integrating and configuring different systems, including peer-to-peer connections between various clouds. Microsoft Entra ID is a cloud solution for identity and access management, serving as a directory and authentication service operating in the cloud. Microsoft Entra ID offers authentication and authorization services for various Microsoft and non-Microsoft services. Within this article, we will explore the integration of the following cloud systems: Slack, MongoAtlas, ElasticCloud, Salesforce, and Datadog. Slack Due to the absence of a precise guide, we decided to create our own, detailing all the headaches that arose along the way. So, let's navigate to the "Enterprise Application" section in Azure. It's necessary to create a new corporate application (this can be done in a few clicks). Next, access the application we've just created and configure Single Sign-On – SAML. Make changes from top to bottom: Identifier: https://slack.com Reply URL: https://<DOMAIN NAME>.slack.com/sso/saml (in case you have ent Grid – https://<DOMAIN NAME>.enterprise.slack.com/sso/saml ) Sign-on URL: https://<DOMAIN NAME>.slack.com/sso/saml/start Logout URL: https:// <DOMAIN NAME>.slack.com/sso/saml/logout Create a "Member" role (or choose another name). Next, create a test user in Microsoft Entra ID and add them to our application. Then, go to the Admin Center in Slack. Follow the steps: Slack Admin - Authentication - SAML configuration - Configure The next step is to configure SAML. We recommend paying attention to the "Display Name" setting. Further, in the settings, uncheck "Update Profile on login," disallow changing the email address and Display Name. There is also an option to customize the text on the login button and prohibit users from logging in using any other method except SSO. After this, you should be able to sign in to Slack using your Microsoft account. If any additional fields or settings are needed, this can be done through Azure AD Provision. To add a new attribute, go to User Attributes and select Add New Mapping. Choose the desired source attribute that you want to be displayed in Slack and associate it with the corresponding Slack attribute. After adding the necessary attributes and saving the settings, proceed to Provision on Demand to test the functionality. Upon a positive outcome, you will receive the following: Next, you need to go to Slack and specify which attributes to display in the complete user profile and where to retrieve them. Go to Profiles and choose the information to be displayed and where it will be pulled from. In this case, SCIM corresponds to Azure AD. Next, we recommend adding the necessary users from Active Directory to Slack to experience the seamless integration. We also want to bring attention to an issue encountered when removing users: if a user is manually deleted from Slack (via the Slack admin center), an error will persist, indicating that the user cannot be transferred to Slack due to insufficient permissions. This occurs because Entra ID retains information about users within the scope and assigns them a unique ID. Deleting and re-adding a user in Slack is perceived as a new user by Entra ID, leading to permission issues during updates. Check the documentation . here MongoAtlas For MongoAtlas, the process follows a similar path. You'll also need to create a new corporate application; for ease, enter "MongoDB Atlas — SSO" in the search field. More detailed instructions can be found here. Next, proceed to the Single Sign-On settings, where you'll need to input the following: In the "Identifier" text field, enter the URL using the following template: https://www.okta.com/saml2/service-provider/<Customer_Unique> In the "Response URL" text field, enter the URL using the template: https://auth.mongodb.com/sso/saml2/<Customer_Unique> In the "Login URL" text field, enter the URL using the template: https://cloud.mongodb.com/sso/<Customer_Unique> We recommend paying attention to the Attributes section and configuring it as shown in the screenshot below (it was essential to add "memberOf"). Next, on the "Setting up Single Sign-On with SAML" page, go to the "SAML Signing Certificate" section and find the XML metadata for federation. Select "Download" to download the certificate and save it locally (Federation Metadata XML). We will need it later in MongoDB Atlas. In the "Setting up MongoDB Atlas — SSO" section, copy the respective URLs. Next, navigate to . MongoDB Atlas -> Organization -> Settings -> Federated Authentication Settings Begin by adding the Microsoft Entra ID credential provider in Atlas. Input the respective URLs mentioned in the previous step from the Microsoft portal , upload . (Issuer URI, Login URL, Single Sign-On URL) the Identity Provider Signature Certificate, and configure the remaining settings as shown in the screenshot below Click "Next" and save the metadata file locally, as you will need to upload it later in Azure on the Single Sign-On configuration page. Next, add, map, and validate the domain, for example, using the corresponding TXT records generated in MongoDB Atlas. After that, link your domain with the credential provider and move on to the most interesting part: role configuration. We've created the following roles as an example, each with specific permissions granted to projects. Then, you need to map these roles to Azure roles. Go to "App roles" for your application, create, and add roles with values exactly matching the names you just assigned to these roles in MongoAtlas. Afterward, add users to the application in the Microsoft Entra ID Enterprise Application and assign them the appropriate roles. Activate "Sign in with Azure SSO" in the Federation Management section in MongoDB Atlas and verify the correct functioning of the configured settings. Refer to the documentation for more detailed instructions. here Elastic Cloud Let's move on to the integration with Elastic Cloud. The fundamental steps for creating an application in the Enterprise Application Entra ID are similar to the previous ones. There, you can find information on where to obtain the Identifier or Logout URL. However, we encountered an issue with attributes, so we'll provide a screenshot of our working configuration. We created two roles - a reader and a superuser. Here, you can create as many roles as you need. Role settings in Elasticsearch are configured in Role Mappings and look as follows: Now comes the interesting part. Two things need to be done: add Azure SSO configuration for Kibana and Elasticsearch. To achieve this, follow these steps: Navigate to the editing section of your Elasticsearch Cloud deployment to update the elasticsearch.yml configuration. Add the following lines: “ xpack: security: authc: realms: saml: saml-azure-ad: order: 2 attributes.principal: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name" attributes.groups: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/role" attributes.name: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname" attributes.mail: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress" idp.metadata.path: "APP Federation metadata URL" idp.entity_id: "Azure Entra ID Identifier" sp.entity_id: "Kibana endpoint" sp.acs: "Kibana endpoint/api/security/v1/saml" sp.logout: "Kibana endpoint/logout" “ After adding and modifying the values highlighted in red to match your configuration, click "Save." Elastic Cloud will validate all changes and initiate the cluster update. This process may take a few minutes. Following the successful completion of the previous step, you need to edit the kibana.yml configuration to allow logins from Microsoft Entra ID in addition to the default login. To do this, add the following lines to the settings: “ xpack.security.authc.providers: saml.saml1: order: 0 realm: saml-azure-ad description: "Log in with Azure Entra ID" basic.basic1: order: 1 ” After completing the previous two steps, another login option should be available on your Kibana login page. Next, proceed with adding users to the application in the Microsoft Entra ID Enterprise Application and assigning them the necessary roles. Refer to the documentation here for more information. Salesforce Integration with Salesforce was a bit more complex than the previous ones. The initial steps are similar, but attention should be given to the Provisioning section. There are nuances regarding specifying admin credentials under which account synchronization will occur, and enabling Attribute Mapping is necessary. Here is an example of our configuration settings in the screenshot. We didn't encounter any significant issues with attributes, but we'll also provide a screenshot of the ones we used. In Salesforce itself, there weren't any particular nuances, and we proceeded with the following steps as outlined in this . guide Datadog During the integration with Datadog, we also encountered some issues. The primary documentation that guided us can be found . For troubleshooting nuances, we referred to guide. here this Everything started smoothly, and initially, it seemed like there wouldn't be any issues. As usual, we created a new Enterprise Application in Microsoft Entra ID. In the "Set up Single Sign-On with SAML" section, pay attention to the location you selected for your Datadog account. In our case, it was the Europe region. As mentioned earlier, for MongoAtlasDB, we add the additional attribute "memberOf" for role mapping. Save all the settings and download the Federation Metadata XML. Upload this file into the SAML settings in Datadog. Here is a convenient for reference. Datadog SAML Settings (replace "EU" with the appropriate domain zone if your location differs). It should look like the screenshot below. link In the organization's Login Methods settings, keep only what is necessary (in my case, I disabled password login and Google Authenticate). Add roles in Azure in the application registration for Microsoft Entra ID. Here comes the most interesting part. Navigate to SAML Group Mappings -> Role Mappings in Datadog and add the key, value, and role. It seems straightforward. See the screenshot below. But it doesn’t work, and it keeps giving the error "There are no AuthNMappings for this user". For quite a while, we tried to understand what we were doing wrong, reviewing all the documentation, logs, and troubleshooting pages. Unfortunately, there was no mention of this anywhere. In the end, the solution is simple: follow the configuration shown in the screenshot below. In the KEY field, enter the string: http://schemas.microsoft.com/ws/2008/06/identity/claims/role After doing this, everything started working successfully. "This is just a small part of the systems that we integrated with Microsoft Entra ID. We hope that the insights shared here will serve as a helpful resource, streamlining your integration processes and saving valuable time. In conclusion, the following advantages of using Microsoft Entra ID can be highlighted: Single Sign-On, which doesn’t require entering the password multiple times; no additional components required; ease of configuration and administration; group policies and groups; device registration; and high-level security.

Navigating Microsoft Entra ID Integrations for Slack, MongoAtlas, Salesforce, and More

স্ল্যাক, মঙ্গোআটলাস, সেলসফোর্স এবং আরও অনেক কিছুর জন্য মাইক্রোসফ্ট এন্ট্রা আইডি ইন্টিগ্রেশন নেভিগেট করা

Navigieren durch Microsoft Entra ID-Integrationen für Slack, MongoAtlas, Salesforce und mehr

Navegando por las integraciones de Microsoft Entra ID para Slack, MongoAtlas, Salesforce y más

Navigation dans les intégrations Microsoft Entra ID pour Slack, MongoAtlas, Salesforce et plus encore

स्लैक, मोंगोएटलस, सेल्सफोर्स और अन्य के लिए माइक्रोसॉफ्ट एंट्रा आईडी एकीकरण को नेविगेट करना

Slack、MongoAtlas、Salesforce などの Microsoft Entra ID 統合のナビゲート

Slack, MongoAtlas, Salesforce 등을 위한 Microsoft Entra ID 통합 탐색

Navegando nas integrações do Microsoft Entra ID para Slack, MongoAtlas, Salesforce e muito mais

Навигация по интеграции Microsoft Entra ID для Slack, MongoAtlas, Salesforce и других

Slack, MongoAtlas, Salesforce ve Daha Fazlası için Microsoft Entra ID Entegrasyonlarında Gezinme

Điều hướng Tích hợp Microsoft Entra ID cho Slack, MongoAtlas, Salesforce, v.v.

导航 Slack、MongoAtlas、Salesforce 等的 Microsoft Entra ID 集成

The Noonification: Robots Driven by Blockchain and Not AI Could Be More Trustworthy (7/9/2024)

DATADOG

datadoghq.com

1K-5K employees

Since 2010

COMPANY RANKING

DATADOG

EVERGREEN INDEX #4886

Representing this company? Click here to claim and customize this page!

Datadog WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

DATADOG

datadoghq.com

1K-5K employees

Since 2010

COMPANY RANKING

DATADOG

EVERGREEN INDEX #4886

RELATED COMPANIESRANK

STORIES ABOUT DATADOG

Representing this company? Click here to claim and customize this page!

Datadog WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!