Skip to main content

CloudWatch Metrics to the OpenTelemetry Collector via Kinesis Firehose

Overview​

This guide streams AWS CloudWatch metrics directly into the OpenTelemetry Collector using the awsfirehosereceiver. A CloudWatch Metric Stream pushes metrics through Amazon Data Firehose (formerly Kinesis Data Firehose), which delivers them over HTTP to the Collector. The Collector decodes the CloudWatch payload and forwards it to base14 Scout over OTLP.

Compared to the Firehose to S3 to Lambda approach, this path has no S3 bucket, no Lambda code to maintain, and lower latency. The trade-off is that the Collector must expose an inbound HTTPS endpoint that Firehose can reach. For a comparison of all approaches, see the AWS CloudWatch overview.

Configuration pending validation

The Collector configuration below is authored from the OpenTelemetry Collector Contrib documentation and has not yet been validated end-to-end against a live AWS account and Scout tenant. Test it in a non-production environment and confirm metrics arrive in Scout before relying on it.

How it works​

CloudWatch ─▶ Metric Stream ─▶ Amazon Data Firehose ─▶ OTel Collector ─▶ Scout
              (JSON format)     (HTTP endpoint)         awsfirehosereceiver   (OTLP)

The awsfirehosereceiver listens on an HTTP port. Firehose posts batches of records to it, authenticating each request with a shared access key sent in the X-Amz-Firehose-Access-Key header. With record_type: cwmetrics, the receiver decodes CloudWatch Metric Stream JSON into OTel metrics.

Prerequisites​

RequirementMinimumRecommended
OTel Collector Contrib0.90.0latest
Public HTTPS endpointvalid CA-signed certACM cert on a load balancer
base14 ScoutAny-
AWS permissionsCloudWatch, Amazon Data Firehose, S3 (backup)-

Before starting:

  • The Collector must be reachable from AWS over HTTPS with a valid certificate. Firehose does not deliver to self-signed certificates.
  • Have your Scout OAuth client ID, client secret, token URL, and OTLP endpoint ready (see Scout Exporter Configuration).
  • Choose a strong shared secret for the Firehose access key.

Step 1: Deploy the Collector with the Firehose receiver​

Create otel-collector-config.yaml. This reuses the Scout oauth2client extension and otlphttp/b14 exporter, and adds the awsfirehose receiver:

otel-collector-config.yaml
extensions:
  oauth2client:
    client_id: ${env:SCOUT_CLIENT_ID}
    client_secret: ${env:SCOUT_CLIENT_SECRET}
    endpoint_params:
      audience: b14collector
    token_url: ${env:SCOUT_TOKEN_URL}
    tls:
      insecure_skip_verify: true

receivers:
  awsfirehose:
    endpoint: 0.0.0.0:4433
    record_type: cwmetrics
    access_key: ${env:FIREHOSE_ACCESS_KEY}
    # Firehose requires HTTPS with a valid certificate. Terminate TLS here,
    # or omit this block and terminate at a load balancer (see below).
    tls:
      cert_file: /etc/otel/certs/server.crt
      key_file: /etc/otel/certs/server.key

processors:
  resource:
    attributes:
      - key: cloud.provider
        value: aws
        action: upsert
      - key: environment
        value: ${env:ENVIRONMENT}
        action: upsert
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  otlphttp/b14:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    auth:
      authenticator: oauth2client
    tls:
      insecure_skip_verify: true

service:
  extensions: [oauth2client]
  pipelines:
    metrics:
      receivers: [awsfirehose]
      processors: [resource, batch]
      exporters: [otlphttp/b14]

Environment variables​

.env
SCOUT_CLIENT_ID=__YOUR_CLIENT_ID__
SCOUT_CLIENT_SECRET=__YOUR_CLIENT_SECRET__
SCOUT_TOKEN_URL=https://id.b14.dev/realms/__YOUR_TENANT__/protocol/openid-connect/token
OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.play.b14.dev/__YOUR_TENANT__/otlp
FIREHOSE_ACCESS_KEY=<a-strong-shared-secret>
ENVIRONMENT=production

TLS termination: the cleanest production pattern is to terminate TLS at a load balancer with an AWS Certificate Manager (ACM) certificate and let the Collector listen on plain HTTP behind it. In that case, omit the tls block on the receiver and point Firehose at the load balancer's HTTPS URL. The X-Amz-Firehose-Access-Key header still passes through, so the access_key check continues to work.

Step 2: Create the Firehose stream (HTTP endpoint delivery)​

  1. Open the Amazon Data Firehose console and click Create Firehose stream.
  2. Set Source to Direct PUT and Destination to HTTP Endpoint.
  3. Under HTTP endpoint details, set:
    • HTTP endpoint URL to your Collector's public HTTPS URL, for example https://collector.example.com:4433 (or the load balancer URL on 443).
    • Access key to the same value as FIREHOSE_ACCESS_KEY.
    • Content encoding to GZIP.
  4. Under Buffer hints, a buffer of 1 MiB or 60 seconds is a reasonable start.
  5. Under Backup settings, select Failed data only and choose an S3 bucket. Firehose requires a backup bucket for records it cannot deliver.
  6. Click Create Firehose stream.

Step 3: Create the CloudWatch Metric Stream​

  1. Open the CloudWatch console, go to Metrics > Streams, and click Create metric stream.
  2. Choose Custom setup with Firehose and select the Firehose stream from Step 2.
  3. Set Output format to JSON to match record_type: cwmetrics.
  4. Under Metrics to be streamed, select the specific namespaces you want (for example AWS/EC2, AWS/RDS) instead of all namespaces to reduce volume and cost.
  5. Give the stream a name and click Create metric stream.

Metrics now flow CloudWatch → Firehose → Collector → Scout.

Verify the setup​

  1. In the Firehose console, open your stream's Monitoring tab and confirm DeliveryToHTTPEndpoint.Success is non-zero and the failed-delivery count is flat.

  2. Check the Collector logs for accepted requests from the receiver:

    docker logs otel-collector | grep -i firehose

  3. In Scout, confirm the CloudWatch metrics appear (metric names are prefixed amazonaws.com/<namespace>/<MetricName>). Allow 5-10 minutes for the first metrics to arrive.

Troubleshooting​

Firehose delivery is failing (records land in the S3 backup)​

Cause: the Collector endpoint is not reachable or its certificate is not valid.

Fix:

  1. Confirm the endpoint is publicly reachable over HTTPS and the certificate is CA-signed, not self-signed.
  2. Verify the port and path in the Firehose HTTP endpoint URL match the receiver's endpoint.
  3. Check the Firehose stream's error logs for the HTTP status returned by the Collector.

Firehose reports 401 or 403 from the Collector​

Cause: the access key does not match.

Fix: confirm the Firehose Access key equals FIREHOSE_ACCESS_KEY on the receiver. The receiver validates the X-Amz-Firehose-Access-Key header.

Requests arrive but no metrics reach Scout​

Cause: a record-type or output-format mismatch, or an export failure.

Fix:

  1. If the Metric Stream output format is OpenTelemetry 1.0.0, set record_type: otlp_v1 on the receiver. Use cwmetrics only with JSON output.
  2. Check the Collector logs for OTLP export errors and confirm the oauth2client values and OTEL_EXPORTER_OTLP_ENDPOINT are correct.

FAQ​

What is the awsfirehosereceiver?

The awsfirehosereceiver is an OpenTelemetry Collector Contrib receiver that accepts records pushed by Amazon Data Firehose over an HTTP endpoint. With record_type set to cwmetrics it decodes CloudWatch Metric Stream JSON directly into OTel metrics, so no S3 bucket or Lambda function is needed.

Do I need a public endpoint for Firehose HTTP delivery?

Yes. Amazon Data Firehose HTTP endpoint delivery requires a publicly reachable HTTPS endpoint with a valid, CA-signed certificate. Terminate TLS on the Collector directly or behind a load balancer with an ACM certificate. Firehose will not deliver to a self-signed certificate.

What output format should the CloudWatch Metric Stream use?

Use JSON output format with record_type: cwmetrics on the receiver. If you set the Metric Stream output format to OpenTelemetry 1.0.0 instead, set record_type: otlp_v1 so the receiver decodes the OTLP payload.

How is this different from the Firehose to S3 to Lambda approach?

This approach delivers straight from Firehose to the Collector's receiver, with no S3 bucket and no Lambda code to maintain, and lower latency. The trade-off is that the Collector must expose an inbound HTTPS endpoint that Firehose can reach, whereas the S3 and Lambda path needs no inbound endpoint.

Was this page helpful?