Skip to main content

AWS ECS

Deploy and configure the Scout Collector on ECS.

Overview​

This guide covers how to collect telemetry data (logs, metrics, and traces) from your ECS environment and send it to base14 Scout.

  • Install base14 Scout's Scout Collector using Task Definitions.
  • Configure telemetry collection for ECS Nodes.
  • Configure custom metrics endpoints
  • Implement trace collection

Prerequisites​

  • An ECS cluster
  • AWS CLI setup with ecs:* permissions.
  • Scout account credentials
    • Endpoint URL
    • API Key
    • Token URL
    • Application Name

Quick Start Guide​

Deploy Scout Collector in minutes by following these steps:

Task Definitions​

For Fargate, you can deploy the Scout collector in different modes:

Best for: Centralized telemetry collection and processing

  • Runs as a standalone REPLICA service
  • Collects telemetry from multiple applications and AWS services
  • Ideal for collecting metrics from RDS, ElastiCache, Amazon MQ, and application traces

Download the required files:

curl -o task-definition.json \
  https://raw.githubusercontent.com/base-14/docs/main/configs/ecs/fargate/task-definition.json

curl -o scout-service-collector-config.yaml \
  https://raw.githubusercontent.com/base-14/docs/main/configs/ecs/fargate/scout-collector-config.yaml
warning
  • This service mode collector can include receivers for AWS services (RDS, ElastiCache, Amazon MQ) and external databases.
  • Review the configuration to add or remove pipelines based on your monitoring needs.
  • Visit docs.base14.io for more details on the configuration.

Generate Configuration​

Replace the placeholders with your actual values:

SERVICE_NAME='<service name>' \
ENVIRONMENT='<environment>' \
SCOUT_ENDPOINT='<scout backend endpoint>' \
SCOUT_CLIENT_ID='<client id>' \
SCOUT_CLIENT_SECRET='<client secret>' \
SCOUT_TOKEN_URL='<token url>' \
envsubst < scout-service-collector-config.yaml > scout-service-collector-config.yaml.tmp && \
mv scout-service-collector-config.yaml.tmp scout-service-collector-config.yaml

Store Configuration in AWS Secrets Manager​

# Create secret for service collector configuration
aws secretsmanager create-secret \
  --name "/ecs/scout/otelcol-service-config" \
  --description "Scout OTEL Service Collector Configuration for Fargate" \
  --secret-string file://scout-service-collector-config.yaml

If the secret already exists, update it:

aws secretsmanager update-secret \
  --secret-id "/ecs/scout/otelcol-service-config" \
  --secret-string file://scout-service-collector-config.yaml

Get Secret ARN​

After creating the secret, retrieve its full ARN (required for task definition):

aws secretsmanager describe-secret \
  --secret-id "/ecs/scout/otelcol-service-config" \
  --query 'ARN' \
  --output text

Save this ARN - you'll need it in the next step.

Generate Task Definition​

Replace the placeholders with your actual values:

AWS_TASK_EXECUTION_ROLE='<task execution role ARN>' \
TASK_NAME='Scout_service_collector' \
SERVICE_NAME='Scout_service_collector' \
SECRET_ARN='<secret ARN from previous step>' \
envsubst < task-definition.json > scout-service-collector-task-definition.json
tip

To find your ECS task execution role ARN:

aws iam list-roles --query 'Roles[?RoleName==`ecsTaskExecutionRole`].Arn' --output text

Register Task Definition​

Register the task definition with ECS:

aws ecs register-task-definition \
  --cli-input-json file://scout-service-collector-task-definition.json

Get Network Configuration​

Fargate requires network configuration. Get your VPC subnets and security groups:

# Get default VPC subnets
aws ec2 describe-subnets \
  --filters "Name=default-for-az,Values=true" \
  --query 'Subnets[*].SubnetId' \
  --output text

# Get the VPC ID from one of the subnets
VPC_ID=$(aws ec2 describe-subnets \
  --subnet-ids <subnet-id-from-above> \
  --query 'Subnets[0].VpcId' \
  --output text)

# Get the default security group for the VPC
aws ec2 describe-security-groups \
  --filters "Name=vpc-id,Values=$VPC_ID" "Name=group-name,Values=default" \
  --query 'SecurityGroups[0].GroupId' \
  --output text

Deploy Service​

Create the ECS service with network configuration:

aws ecs create-service \
  --cluster <cluster-name> \
  --service-name scout-service-collector \
  --task-definition Scout_service_collector:1 \
  --scheduling-strategy REPLICA \
  --desired-count 1 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[<subnet-id-1>,<subnet-id-2>,<subnet-id-3>],securityGroups=[<security-group-id>],assignPublicIp=ENABLED}"
warning
  • Replace <cluster-name> with your ECS cluster name
  • Replace subnet IDs with the values from the previous step
  • Replace security group ID with the value from the previous step
  • Ensure all subnets belong to the same VPC as the security group

Verify Deployment​

Check the service status:

aws ecs describe-services \
  --cluster <cluster-name> \
  --services scout-service-collector \
  --query 'services[0].{Name:serviceName,Status:status,Running:runningCount,Desired:desiredCount}' \
  --output table

Wait for the task to reach RUNNING status (may take 1-2 minutes):

# List running tasks
aws ecs list-tasks \
  --cluster <cluster-name> \
  --service-name scout-service-collector \
  --desired-status RUNNING

# Check task details
aws ecs describe-tasks \
  --cluster <cluster-name> \
  --tasks <task-arn-from-above> \
  --query 'tasks[0].{Status:lastStatus,Health:healthStatus,Container:containers[0].name}' \
  --output table

If the task stops or fails, check the stopped reason:

aws ecs describe-tasks \
  --cluster <cluster-name> \
  --tasks <task-arn> \
  --query 'tasks[0].stoppedReason'

Update IAM Permissions​

Your ECS Task Execution Role needs permission to access Secrets Manager.

First, create the IAM policy document:

cat > /tmp/secrets-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": [
        "arn:aws:secretsmanager:<aws-region>:<aws-account-id>:secret:/ecs/scout/otelcol-service-config*",
        "arn:aws:secretsmanager:<aws-region>:<aws-account-id>:secret:/ecs/scout/otelcol-sidecar-config*"
      ]
    }
  ]
}
EOF

Replace <aws-region> and <aws-account-id> with your values:

  • Region: The AWS region where you created the secret (e.g., us-east-1)
  • Account ID: Your 12-digit AWS account ID

Then attach the policy to your task execution role:

aws iam put-role-policy \
  --role-name ecsTaskExecutionRole \
  --policy-name ScoutSecretsAccess \
  --policy-document file:///tmp/secrets-policy.json
Common Error

If you skip this step, your tasks will fail with: ResourceInitializationError: unable to retrieve secrets from ssm

This happens because the task execution role cannot access the secret stored in Secrets Manager.

That's it, you're done! Go to the Scout Dashboards to see the data flowing.

Was this page helpful?