AWS ALB Monitoring with OpenTelemetry - Application Load Balancer Metrics

Overview

This guide covers collecting Application Load Balancer metrics (request counts, response times, HTTP status codes, and target health) via CloudWatch Metrics Stream, plus access logs through a Lambda forwarder. We recommend CloudWatch Metrics Stream over Prometheus exporters: it needs no per-service exporter and integrates natively with AWS.

What You'll Monitor

ALB monitoring combines two data sources - CloudWatch metrics for the aggregate picture and access logs for per-request detail:

CloudWatch Metrics Stream (AWS/ApplicationELB):

Metric	What it tells you
RequestCount	Total requests the load balancer processed
TargetResponseTime	Time targets took to respond (watch p90/p99)
HTTPCode_Target_2XX_Count / 4XX / 5XX	Response codes returned by your targets
HTTPCode_ELB_4XX_Count / 5XX_Count	Errors generated by the load balancer itself
HealthyHostCount / UnHealthyHostCount	Targets passing / failing health checks
ActiveConnectionCount	Concurrent connections through the load balancer
NewConnectionCount	New connections established per period
RejectedConnectionCount	Connections dropped after hitting the connection limit
TargetConnectionErrorCount	Connections that failed to reach a target
ProcessedBytes	Total bytes processed (request and response)
ConsumedLCUs	Capacity units consumed - the main cost driver
RequestCountPerTarget	Average requests per target (scaling signal)

Access logs (per-request fields via the Lambda forwarder):

Field	What it tells you
elb_status_code	Status the load balancer returned to the client
target_status_code	Status the target returned to the load balancer
target_processing_time	Seconds the target took to process the request
request_processing_time / response_processing_time	Time spent inside the load balancer
client:port	Client IP and port that made the request
target:port	Target IP and port that served it
request	Method, URL, and HTTP version
user_agent	Client user agent string
ssl_cipher / ssl_protocol	TLS cipher and protocol negotiated
trace_id	X-Amzn-Trace-Id for correlating across hops

Prerequisites

Requirement	Minimum	Recommended
Load balancer	Application Load Balancer	ALB with access logs enabled
OTel Collector Contrib	0.90.0	latest
base14 Scout	Any	-
AWS permissions	CloudWatch, Amazon Data Firehose, S3, Lambda	-

Before starting:

• An Application Load Balancer serving traffic. This guide covers the AWS/ApplicationELB namespace, not Network or Classic load balancers.
• CloudWatch Metrics Stream infrastructure set up (see Step 1).
• For access logs: an S3 bucket with ALB access logging enabled and the Lambda forwarder built below.

Collecting Application ELB Metrics

For collecting Application ELB metrics, we recommend using CloudWatch Metrics Stream instead of Prometheus exporters. CloudWatch Metrics Stream provides:

• Faster delivery: 3-5 minutes end-to-end vs 5+ minutes with polling.
• Lower cost: No need to run dedicated exporters.
• Better scalability: Native AWS service integration.
• Automatic metric discovery: No need to manually configure metric lists.

Step 1: Set up CloudWatch Metrics Stream

Follow our comprehensive CloudWatch Metrics Stream guide to set up the infrastructure.

Step 2: Configure Application ELB metrics filtering

When configuring your CloudWatch Metrics Stream in Step 3 of the setup guide, make sure to:

1. Select specific namespaces instead of "All namespaces"
2. Choose only AWS/ApplicationELB from the namespace list
3. This ensures you only collect Application ELB metrics, reducing costs and data volume

Note: CloudWatch Metrics Stream will automatically deliver all AWS/ApplicationELB metrics including request counts, response times, HTTP status codes, target health, connection counts, and more.

Collecting Application ELB Logs

Before wiring up the Lambda, enable access logging on the load balancer so log files land in S3. In the EC2 console, open your load balancer, go to Attributes > Edit, turn on Access logs, and point them at an S3 bucket. AWS also needs a bucket policy allowing the ELB log-delivery account to write there; the console offers to create it. Without this, the bucket stays empty and the Lambda never fires.

Step 1: Creating a Lambda function

1. Go to your AWS console and search for AWS Lambda, go to Functions and click on Create Function.
2. Choose the Author from scratch checkbox and proceed to fill in the function name.
3. Choose Python 3.x as the Runtime version, x86_64 as Architecture (preferably), and keep other settings as default. Select Create a new role with basic Lambda permissions for now, we'll require more permissions later. So for now, select this option.
4. Once you are done configuring the Lambda function, your Lambda function is created.

Step 2: Configuring Policies for Lambda function

As said in previous step, we need extra permissions in order to access the S3 Bucket for execution of our Lambda code, follow along to set it up.

1. Scroll down from your Lambda page, you'll see a few tabs there. Go to Configurations and select Permissions from the left sidebar.
2. Click on the Execution Role name link just under Role name to open the role in the AWS IAM console.
3. Click Add permissions > Create inline policy, switch to the JSON editor, paste the policy below (scoped to your log bucket only), then name and create it.

lambda-s3-read-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-alb-logs-bucket/*"
    }
  ]
}

Replace your-alb-logs-bucket with your bucket name. The function only needs to read log objects, so avoid broad policies like AmazonS3FullAccess.

Step 3: Adding Triggers

1. Navigate to the Lambda function we just created.
2. Click on the + Add trigger button from the Lambda console.
3. Select S3 from the first drop down of AWS services list. Pick your S3 bucket for the second field.
4. For the Event types field, you can select any number of options you wish. The trigger will occur depending upon what option(s) you choose here. By default, the All object create events will be selected.
5. Verify the settings and click on Add button at bottom right to add this trigger.

Step 4: Adding Request Layer

We will be using python's request module which is not included by default in Lambda.

# create the layer directory (Lambda expects dependencies under python/)
mkdir -p python

# install the requests module into it
pip install --target python requests

# zip it with python/ at the archive root
zip -r dependencies.zip python

1. Run the above commands to create a zip of the request module and add it as a layer to make it work on AWS Lambda.
2. To upload your zip file, go to AWS Lambda > Layers and click on Create Layer. [Not inside your specific Lambda function, just the landing page of AWS Lambda].
3. You'll be redirected to Layer configurations page. Here, give a name to your layer, an optional description, select Upload a .zip file, click on Upload and locate the dependencies.zip file.
4. Select the same architecture and Python runtime as the function - the layer must match, or the requests import fails at runtime. Hit Create to build the layer.
5. Go to your Lambda function, scroll down to Layers section and on the right of it, you'll find a button that says Add a layer to click on.
6. Pick Custom layers from the checkbox and select your custom layer from the given drop down below and then click on the button Add.

Step 5: The Lambda Function

The Lambda function reads gzipped ALB access-log files from S3, converts each log line to OTLP JSON, and posts the result to your Collector endpoint.

import json
import gzip
import boto3
import requests
import shlex
import os
from datetime import datetime

# Create an S3 client
s3 = boto3.client('s3')
client_id=os.environ.get('CLIENT_ID')
client_secret=os.environ.get('CLIENT_SECRET')
token_url=os.environ.get('TOKEN_URL')
endpoint_url=os.environ.get('ENDPOINT_URL')

# Function to convert a log line into a JSON object
def convert_log_line_to_json(line):
 # Define the headers to be used for the JSON keys (ALB log format)
 headers = ["type", "time", "elb", "client:port", "target:port", "request_processing_time",
            "target_processing_time", "response_processing_time", "elb_status_code",
            "target_status_code", "received_bytes", "sent_bytes", "request", "user_agent",
            "ssl_cipher", "ssl_protocol", "target_group_arn", "trace_id", "domain_name",
            "chosen_cert_arn", "matched_rule_priority", "request_creation_time",
            "actions_executed", "redirect_url", "error_reason", "target:port_list",
            "target_status_code_list", "classification", "classification_reason"]

 # Split the log line using shell-like syntax (keeping quotes, etc.)
 parts = shlex.split(line, posix=False)

 # Create a dictionary with as many pairs as possible
 result = {}
 for i in range(min(len(headers), len(parts))):
  result[headers[i]] = parts[i]

 return result


# Convert logs to OTLP format
def convert_to_otlp_format(logs):
 current_time_ns = int(datetime.now().timestamp() * 1_000_000_000)  # nanoseconds

 # Create OTLP log records
 resource_logs = {
  "resourceLogs": [{
   "resource": {
    "attributes": [
     {"key": "service.name", "value": {"stringValue": "alb"}},
     {"key": "cloud.provider", "value": {"stringValue": "aws"}},
     {"key": "environment", "value": {"stringValue": os.environ.get('ENVIRONMENT', 'production')}}
    ]
   },
   "scopeLogs": [{
    "scope": {},
    "logRecords": []
   }]
  }]
 }

 # Add each log entry as a log record
 for log in logs:
  # Create attributes from log fields
  attributes = []
  for key, value in log.items():
   attributes.append({
    "key": key,
    "value": {"stringValue": value}
   })

  # Get timestamp if available, or use current time
  timestamp = current_time_ns
  if "time" in log:
   try:
    # Try to parse the ALB log timestamp format
    dt = datetime.strptime(log["time"], "%Y-%m-%dT%H:%M:%S.%fZ")
    timestamp = int(dt.timestamp() * 1_000_000_000)
   except (ValueError, TypeError):
    pass

  # Create a log record
  log_record = {
   "timeUnixNano": timestamp,
   "severityText": "INFO",
   "body": {"stringValue": json.dumps(log)},
   "attributes": attributes
  }

  resource_logs["resourceLogs"][0]["scopeLogs"][0]["logRecords"].append(log_record)

 return resource_logs


# Lambda function handler
def lambda_handler(event, context):
 try:
  # Check if this is being triggered by an S3 event
  if 'Records' in event and event['Records'][0].get('eventSource') == 'aws:s3':
   # Get the S3 bucket and key from the event
   s3_event = event['Records'][0]['s3']
   bucket_name = s3_event['bucket']['name']
   file_key = s3_event['object']['key']

   # Only process log files
   if not file_key.endswith('.log.gz'):
    print(f"Skipping non-log file: {file_key}")
    return {
     'statusCode': 200,
     'body': 'Skipped non-log file'
    }

   log_files = [file_key]
  else:
    print(f"Manual Trigger is not supported yet")
    return {
      'statusCode': 403,
      'body': 'Manual Trigger is not supported yet'
      }


  processed_files = 0
  total_logs = 0

  # Process each log file
  for file_key in log_files:
   print(f"Processing file: {bucket_name}/{file_key}")

   # Download the gzipped file content
   file_obj = s3.get_object(Bucket=bucket_name, Key=file_key)
   file_content = file_obj['Body'].read()

   # Decompress the gzipped content
   decompressed_content = gzip.decompress(file_content)

   # Convert bytes to string
   log_text = str(decompressed_content, encoding='utf-8')

   # Split the string into lines and filter out empty lines
   lines = [line for line in log_text.strip().split('\n') if line.strip()]

   log_count = len(lines)
   print(f"File contains {log_count} log entries")

   # Process logs in batches to prevent timeouts
   batch_size = int(os.environ.get('BATCH_SIZE', '100'))
   for i in range(0, log_count, batch_size):
    batch_lines = lines[i:min(i + batch_size, log_count)]

    # Convert each log line string into a JSON object
    json_logs = [convert_log_line_to_json(line) for line in batch_lines]

    # Convert to OTLP format
    otlp_data = convert_to_otlp_format(json_logs)

    # Set headers for OTEL collector
    headers = {
     'Content-Type': 'application/json'
    }

    http_url = f"{endpoint_url}/v1/logs"

    token_response = requests.post(
        token_url,
        data={
            "grant_type": "client_credentials",
            "audience": "b14collector",
            },
        auth=(client_id, client_secret),
    )
    token_response.raise_for_status()
    access_token = token_response.json()["access_token"]
    headers["Authorization"] = f"Bearer {access_token}"


    # Send the JSON data to the OTEL collector
    try:
     response = requests.post(http_url, json=otlp_data, headers=headers,
                              timeout=float(os.environ.get('REQUEST_TIMEOUT', '5')))
     response.raise_for_status()
     print(f"Sent batch of {len(batch_lines)} logs to {http_url}. Response: {response.status_code}")
    except requests.exceptions.RequestException as e:
     print(f"Error sending logs to OTEL collector: {str(e)}")
     if hasattr(e, 'response') and e.response:
      print(f"Response status: {e.response.status_code}")
      print(f"Response body: {e.response.text[:200]}...")

   total_logs += log_count
   processed_files += 1

  return {
   'statusCode': 200,
   'body': f'Successfully processed {processed_files}:{total_logs} log entries'
  }

 except Exception as e:
  print(f"Error processing logs: {str(e)}")
  import traceback
  traceback.print_exc()
  return {
   'statusCode': 500,
   'body': f'Error: {str(e)}'
  }

Set the Lambda's environment variables: CLIENT_ID, CLIENT_SECRET, TOKEN_URL, ENDPOINT_URL (your Scout OTLP base endpoint - the function appends /v1/logs), and ENVIRONMENT (for example production).

Verify the setup

After deploying, send traffic through the load balancer, then confirm both paths are flowing:

1. In Scout, check for the CloudWatch metrics, named amazonaws.com/AWS/ApplicationELB/<MetricName> (request counts, response times, HTTP status codes).
2. Wait for an access-log object to land in the S3 bucket, then open the Lambda's CloudWatch log group - a successful run logs Sent batch of N logs ... Response: 200.
3. In Scout, check for logs with service.name = alb.

Allow 5-10 minutes for the first metrics to arrive through the stream.

Key alerts to configure

Once telemetry is flowing, set up alerts on these thresholds:

Metric	Warning	Critical	Why
HTTPCode_ELB_5XX_Count	> 0 sustained	spiking	The load balancer itself is failing requests
HTTPCode_Target_5XX_Count	> 1% of requests	> 5% of requests	Targets are returning server errors
TargetResponseTime (p99)	> 1s	> 3s	Backend latency is degrading user experience
UnHealthyHostCount	> 0	half the targets or more	Targets are failing health checks
RejectedConnectionCount	> 0	sustained	Connection limit reached (LCU saturation)
TargetConnectionErrorCount	> 0	sustained	Load balancer cannot reach its targets

CloudWatch Metrics Stream adds 3-5 minutes of latency, so treat these as trend and capacity alerts, not sub-minute failure detection. The access logs add the per-request detail - client IP, target, elb_status_code, processing times - to debug the 5xx spikes the metrics surface.

Troubleshooting

ALB metrics not appearing in Scout

Cause: Metrics Stream isn't active or isn't filtered to the ELB namespace.

Fix:

1. In CloudWatch > Metrics > Streams, verify the stream is active
2. Confirm the namespace filter includes AWS/ApplicationELB
3. Check that Firehose delivery is succeeding (look at the S3 error prefix)
4. Allow 5-10 minutes for initial metrics to flow

Access logs never reach Scout

Cause: Access logging is off, or the Lambda isn't triggered.

Fix:

1. Confirm access logging is enabled on the load balancer and objects are landing in the S3 bucket
2. Verify the bucket policy grants the ELB log-delivery account write access
3. Check the S3 trigger fires the Lambda on object-create events
4. Read the Lambda's CloudWatch logs for errors

Lambda fails with "No module named requests"

Cause: The requests layer isn't attached, or its architecture doesn't match.

Fix:

1. Attach the dependencies.zip layer to the function
2. Ensure the dependencies sit under python/ at the archive root
3. Confirm the layer's architecture and Python runtime match the function

Lambda gets 401 or 403 from the Collector

Cause: Wrong OAuth2 credentials or endpoint.

Fix:

1. Verify CLIENT_ID, CLIENT_SECRET, and TOKEN_URL
2. Confirm ENDPOINT_URL is the base OTLP endpoint (the function appends /v1/logs)
3. Check the token audience is b14collector

FAQ

How do I monitor AWS ALB with OpenTelemetry?

Use CloudWatch Metrics Stream to collect AWS/ApplicationELB metrics with 3-5 minute end-to-end latency, then forward them through an OpenTelemetry Collector to base14 Scout for visualization and alerting.

What ALB metrics does CloudWatch Metrics Stream provide?

All AWS/ApplicationELB metrics - request counts, target response times, HTTP status codes (ELB and target), healthy/unhealthy host counts, connection counts, and consumed LCUs.

How do I collect AWS ALB access logs with OpenTelemetry?

Enable access logging on the load balancer, then trigger a Lambda on S3 object-create events. The Lambda reads each gzipped log file, converts entries to OTLP, and forwards them to your Collector endpoint.

Should I use CloudWatch Metrics Stream or Prometheus for ALB monitoring?

CloudWatch Metrics Stream is recommended: faster delivery (3-5 min end-to-end vs 5+ min), no dedicated exporter to run, and automatic metric discovery for AWS services.

How do I filter ALB metrics in CloudWatch Metrics Stream?

When configuring the stream, select specific namespaces and choose only AWS/ApplicationELB. This keeps costs and data volume down in Scout.

How do I set up alerts for AWS ALB?

Route AWS/ApplicationELB metrics through CloudWatch Metrics Stream to Scout, then alert on sustained HTTPCode_ELB_5XX_Count, HTTPCode_Target_5XX_Count above 1% of requests, TargetResponseTime p99 above 1s, UnHealthyHostCount above zero, and rising RejectedConnectionCount or TargetConnectionErrorCount.

Related Guides

• Set up AWS metrics streaming with CloudWatch Metrics Stream Setup.
• RDS Monitoring - Monitor AWS RDS databases
• ElastiCache Monitoring - Monitor Redis and Memcached
• Docker Compose Setup - Set up collector for local development