Skip to main content

AWS CloudWatch to base14 Scout - Push and Pull Approaches Compared

Overview​

This is the architectural landing page for getting AWS CloudWatch metrics into base14 Scout. AWS resources publish their operational metrics to CloudWatch (AWS/EC2, AWS/RDS, AWS/ApplicationELB, and so on). There are four supported ways to get that data to Scout, split across push and pull mechanisms. This page frames the trade-offs and points you at the right per-approach guide.

The reader profile is DevOps and SRE engineers who run AWS workloads and want to choose an ingestion path before configuring it. For execution, jump to the guides linked in the approach guides table below.

TL;DR

There is no single best path - pick by your latency, cost, and infrastructure constraints.

Push vs pull​

CloudWatch metrics reach Scout by one of two mechanisms:

  • Push - AWS sends metrics out as they are published. CloudWatch Metric Streams deliver continuously through Amazon Data Firehose (formerly Kinesis Data Firehose) with 2-3 minute latency. This is the freshest data and it lands in the Scout data lake, but it needs streaming infrastructure on the AWS side.
  • Pull - something polls the CloudWatch API on an interval and either ships the result to Scout or renders it directly. Pull is simpler to reason about and needs no inbound endpoint, but it adds the poll interval on top of CloudWatch's own metric-availability delay and it bills against the CloudWatch query APIs.

The two push approaches differ only in what carries the stream to the Collector. The two pull approaches differ in whether the data is stored in Scout at all.

Architecture at a glance​

Approach 1 - Firehose to the Collector (push):

CloudWatch ─▶ Metric Stream ─▶ Amazon Data Firehose ─▶ OTel Collector ─▶ Scout
              (JSON format)     (HTTP endpoint)         awsfirehosereceiver   (OTLP)

Approach 2 - Firehose to S3 to Lambda (push):

CloudWatch ─▶ Metric Stream ─▶ Firehose ─▶ S3 ─▶ Lambda ─▶ Scout
              (JSON format)                       (OTLP forwarder)  (OTLP)

Approach 3 - Prometheus CloudWatch exporter (pull):

                       poll GetMetricData
OTel Collector ◀─ scrape /metrics ─ CloudWatch exporter ──────────────▶ CloudWatch
   │                                                                       API
   └─ OTLP ─▶ Scout

Approach 4 - CloudWatch datasource (pull, no ingestion):

Scout ─ query at render time ─▶ CloudWatch API
  │
  └─ renders panels directly; nothing stored in Scout

Choosing an approach​

ApproachMechanismData pathLatencyStored in Scout?Main cost driverBest for
1. Firehose to CollectorPushMetric Stream → Firehose → awsfirehosereceiver → OTLPStream delivery (~2-3 min)YesFirehose ingestion + Scout ingestLowest-latency push, no custom code, when you can run a public HTTPS collector
2. Firehose to S3 to LambdaPushMetric Stream → Firehose → S3 → Lambda → OTLPStream + S3/Lambda hop (~3-5 min)YesFirehose + S3 + Lambda + Scout ingestPush without an inbound endpoint; already documented and deployed
3. Prometheus exporterPullExporter polls CloudWatch → Collector scrapes → OTLPScrape interval + CloudWatch delayYesCloudWatch GetMetricData API + Scout ingestKubernetes-native scrape model, fine-grained metric selection, no inbound endpoint
4. CloudWatch datasourcePullScout queries CloudWatch at render timeLive at query timeNoCloudWatch GetMetricData APIVisualizing CloudWatch without ingesting or retaining in Scout

Decision guide​

Choose Firehose to the Collector when:

  • You want the freshest CloudWatch data stored in Scout and can expose a publicly reachable HTTPS collector endpoint.
  • You prefer a native Collector receiver over maintaining Lambda code.

Choose Firehose to S3 to Lambda when:

  • You cannot expose an inbound collector endpoint to Firehose.
  • You want S3 to buffer and retain the raw stream, or you already run this pipeline and do not want to change it.

Choose the Prometheus CloudWatch exporter when:

  • You already run a Prometheus-style scrape model, especially in Kubernetes.
  • You want precise control over which namespaces, metrics, and dimensions are collected, and latency is not your first concern.
  • You cannot or do not want to expose an inbound endpoint - the collector pulls, so nothing inbound is required.

Choose the CloudWatch datasource when:

  • You only need to visualize CloudWatch metrics and do not need them stored or retained in Scout.
  • You want the fastest possible setup and are comfortable that the data cannot be correlated with your OTLP telemetry.

The four approaches​

1. Firehose to the OpenTelemetry Collector (push)​

How it works: a CloudWatch Metric Stream sends metrics through Amazon Data Firehose, which delivers them over HTTP to the Collector's awsfirehosereceiver. The Collector forwards to Scout over OTLP.

Best for: the lowest-latency push path that stores data in Scout, without writing or maintaining any Lambda code.

Trade-offs: Firehose HTTP endpoint delivery requires a publicly reachable HTTPS endpoint with a valid certificate, plus a shared access key.

Guide → Firehose to the OTel Collector

2. Firehose to S3 to Lambda (push)​

How it works: the Metric Stream delivers through Firehose to an S3 bucket, and an S3-triggered Lambda parses each object, converts it to OTLP, and posts it to Scout.

Best for: a push pipeline when you cannot expose an inbound collector endpoint, or when you want S3 to buffer the raw stream. This is the approach Scout has documented and deployed the longest.

Trade-offs: more moving parts (S3 and Lambda), an extra hop of latency, and custom Lambda code to maintain.

Guide → CloudWatch Metrics Stream

3. Prometheus CloudWatch exporter (pull)​

How it works: the Prometheus CloudWatch exporter polls the CloudWatch GetMetricData API and exposes the result at /metrics. The Collector's Prometheus receiver scrapes that endpoint and forwards to Scout over OTLP.

Best for: teams that already run a Prometheus scrape model, want fine-grained control over which metrics are collected, and prefer a pull model with no inbound endpoint.

Trade-offs: latency is the scrape interval plus CloudWatch's own delay, and you pay per GetMetricData call. You also run and operate the exporter.

Guide → Prometheus CloudWatch exporter

4. CloudWatch datasource (pull, no ingestion)​

How it works: Scout queries CloudWatch directly when a dashboard loads. base14 enables the CloudWatch datasource for your tenant; you configure AWS credentials and a default region, then build panels against it.

Best for: visualizing CloudWatch metrics with the least setup, when you do not need the data stored or retained in Scout.

Trade-offs: nothing is stored in Scout, so the metrics cannot be joined with your OTLP telemetry and are bound by CloudWatch retention. Each dashboard load bills against the CloudWatch query APIs.

Guide → CloudWatch datasource

Approach guides​

Each guide is the execution playbook for one approach, with the exact AWS setup, the Collector or datasource configuration, and verification steps.

ApproachMechanismGuide
Firehose to the OTel CollectorPushFirehose receiver guide
Firehose to S3 to LambdaPushMetrics stream guide
Prometheus CloudWatch exporterPullPrometheus exporter guide
CloudWatch datasourcePullCloudWatch datasource guide

The per-service AWS guides (RDS, ELB, ElastiCache, Amazon MQ) all build on the push mechanism and delegate the streaming setup to these approach guides.

Frequently Asked Questions​

What are the ways to get AWS CloudWatch metrics into base14 Scout?​

Four. Two push: CloudWatch Metric Streams through Amazon Data Firehose into the OpenTelemetry Collector's awsfirehosereceiver, or the same stream through Firehose to S3 with a Lambda that forwards OTLP. Two pull: the Prometheus CloudWatch exporter scraped by the Collector, or the CloudWatch datasource that queries CloudWatch directly at dashboard render time without ingesting anything.

Which approach has the lowest latency?​

The push approaches. CloudWatch Metric Streams deliver in 2-3 minutes, so the Firehose-to-Collector path is the lowest-latency option that stores data in Scout. Pulling with the Prometheus exporter adds your scrape interval on top of CloudWatch's own metric-availability delay, so it is typically slower.

Which approach avoids ingestion and storage cost in Scout?​

The CloudWatch datasource. It queries CloudWatch directly when a dashboard loads, so nothing is ingested or stored in Scout. You pay CloudWatch query API costs instead, and the data is bound by CloudWatch retention rather than kept long-term in Scout.

Do I need a public OpenTelemetry Collector to receive CloudWatch metrics?​

Only for the Firehose-to-Collector push approach. Amazon Data Firehose HTTP endpoint delivery requires a publicly reachable HTTPS endpoint with a valid certificate. The Firehose-to-S3-to-Lambda path and the Prometheus exporter path do not need an inbound endpoint because the collector or Lambda reaches out to AWS.

Can I query CloudWatch metrics in Scout without ingesting them?​

Yes. Enable the CloudWatch datasource in Scout and query CloudWatch directly from dashboards. base14 provisions the datasource for your tenant. This is query federation, not ingestion, so the metrics are not stored in Scout and cannot be joined with your OTLP telemetry.

Push or pull for CloudWatch metrics?​

Push (CloudWatch Metric Streams through Firehose) when you want the lowest latency and durable storage in Scout. Pull (Prometheus exporter) when a Kubernetes-native scrape model and fine-grained metric selection matter more than latency, or when you cannot expose an inbound endpoint. Use the CloudWatch datasource when you only need to visualize CloudWatch and do not need to retain the data in Scout.

Was this page helpful?