Skip to main content

GCP API Gateway and Self-Managed nginx Monitoring with OpenTelemetry

"API gateway" on GCP means one of two unrelated things, and they are instrumented completely differently. This guide covers both: Google's managed API Gateway service, and a self-managed nginx gateway running on GKE or Compute Engine.

Read GCP Monitoring overview first — it covers the collector image, IAM, and resource attributes this guide assumes.

API Gateway is not nginx

Google Cloud API Gateway is built on Envoy by way of ESPv2, not nginx. If you are looking for nginx instrumentation because you run nginx, skip to Part B. If you use Google's managed service, Part A applies and nginx is irrelevant to you. The two parts of this page do not overlap.

Running this in production

Storing and querying this telemetry at production volume is what base14 Scout does. Check out Scout Metrics.

Choosing your part​

You runReadTelemetry source
Google Cloud API Gateway (managed, ESPv2/Envoy)Part ACloud Monitoring + Cloud Logging
ingress-nginx on GKEPart BPrometheus scrape + filelog
nginx on Compute EnginePart BPrometheus scrape + filelog + OTel module
Envoy or Traefik directlyEnvoy, TraefikNative OTel support

Part A: GCP API Gateway​

API Gateway publishes a small, fixed metric set to Cloud Monitoring under apigateway.googleapis.com/. The monitored resource is apigateway.googleapis.com/Gateway, with labels for gateway_id, location and project_id. Metrics additionally carry api_config, which is the dimension to group by during a config rollout, and response_code_class.

Confirm each metric name in Monitoring → Metrics Explorer with the Active toggle enabled before relying on it — API Gateway's published metric set has changed more than most.

Receiver configuration​

api-gateway-config.yaml
receivers:
  # ...your existing receivers...
  googlecloudmonitoring/apigateway:
    collection_interval: 60s
    project_id: ${env:GCP_PROJECT_ID}
    metrics_list:
      - metric_name: "apigateway.googleapis.com/request_count"
      # Distributions — collector v0.129.0 or later only
      - metric_name: "apigateway.googleapis.com/request_latencies"
      - metric_name: "apigateway.googleapis.com/request_sizes"
      - metric_name: "apigateway.googleapis.com/response_sizes"

processors:
  resource/apigateway:
    attributes:
      - {key: service.name, value: apigateway-metrics, action: insert}
      - {key: cloud.provider, value: gcp, action: insert}
      - {key: cloud.platform, value: gcp_api_gateway, action: insert}
      - {key: cloud.account.id, value: "${env:GCP_PROJECT_ID}", action: insert}
      - {key: deployment.environment.name, value: "${env:ENVIRONMENT}", action: upsert}
      - {key: environment, value: "${env:ENVIRONMENT}", action: upsert}

  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
    check_interval: 5s

  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  otlphttp/base14:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}

service:
  pipelines:
    # ...your existing pipelines...
    metrics/apigateway:
      receivers: [googlecloudmonitoring/apigateway]
      processors: [memory_limiter, resource/apigateway, batch]
      exporters: [otlphttp/base14]
Three of these four are distributions

request_latencies, request_sizes and response_sizes are all DELTA + DISTRIBUTION, which needs collector v0.129.0 or later. On an older build they produce invalid data points that fail the entire scrape batch, so request_count disappears along with them.

If you cannot upgrade, delete the three marked lines and collect request_count alone. That leaves you with error rate and traffic volume, and no latency at all — API Gateway publishes latency only as a distribution.

Environment variables​

.env
GCP_PROJECT_ID=your-gcp-project-id
ENVIRONMENT=production
OTEL_EXPORTER_OTLP_ENDPOINT=https://<your-tenant>.base14.io

# Not needed if using GKE Workload Identity
GOOGLE_APPLICATION_CREDENTIALS=/path/to/scout-telemetry-reader-key.json

What you'll monitor​

MetricKindUnitUse case
request_countDelta sumcountRequest rate, labelled by response_code and response_code_class.
request_latenciesDelta histogrammsEnd-to-end gateway latency, including the backend.
request_sizesDelta histogrambytesPayload sizes; where request-size limit rejections originate.
response_sizesDelta histogrambytesResponse volume, and egress cost.

That is the whole set. API Gateway exposes no per-route breakdown in metrics, no backend-versus-gateway latency split, and no authentication outcome dimension. For any of those you need the logs.

Logs​

API Gateway request logs go to Cloud Logging under resource.type="apigateway.googleapis.com/Gateway". Route them through the sink you set up in GCP Cloud Logging:

gcloud logging sinks create scout-apigateway-logs \
  pubsub.googleapis.com/projects/PROJECT_ID/topics/scout-logs \
  --log-filter='resource.type="apigateway.googleapis.com/Gateway"'

Entries carry an httpRequest block, which the google_cloud_logentry_encoding extension maps to http.request.method, url.full, http.response.status_code, http.request.server.duration and network.peer.address. The API config and route that matched appear in the payload.

If your backend runs ESPv2 on Cloud Run directly rather than behind API Gateway, it also publishes under serviceruntime.googleapis.com/api/ — a richer set including per-method breakdowns and quota metrics.

Alert tuning​

SignalSource metricWarningCritical
Error raterequest_count where response_code_class = 500> 1% for 5m> 5% for 5m
Client errorsrequest_count where response_code_class = 400> 10% for 15m> 25% for 15m
Latencyrequest_latencies p99> 1s for 10m> 3s for 5m
Traffic stoprequest_countzero for 10mzero for 30m

A high 4xx rate on API Gateway usually means authentication or schema validation rejections rather than genuine client bugs — the gateway rejects before the backend sees anything. The logs distinguish them.

Part B: Self-managed nginx​

nginx exposes four numbers by default, through the stub_status module. Getting useful gateway telemetry means choosing among three sources, which stack.

SourceGives youCost
nginx receiver via stub_status4 connection and request countersTrivial
prometheus receiver via nginx-prometheus-exporterPer-upstream, per-status detailOne extra process
filelog receiver on the access logPer-request detail, any field you logLog volume
nginx-module-otelReal distributed tracesA module build

The receiver configuration for all four is documented once in nginx component. This section covers only what changes when nginx is a GCP API gateway.

ingress-nginx on GKE​

ingress-nginx already exposes Prometheus metrics on port 10254 — no exporter needed. Scrape it with the prometheus receiver:

nginx-gateway-config.yaml
receivers:
  # ...your existing receivers...
  prometheus/nginx:
    config:
      scrape_configs:
        - job_name: ingress-nginx
          scrape_interval: 30s
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names: [ingress-nginx]
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_container_port_name]
              action: keep
              regex: metrics

processors:
  resource/nginx:
    attributes:
      - {key: service.name, value: nginx-gateway-metrics, action: insert}
      - {key: cloud.provider, value: gcp, action: insert}
      - {key: cloud.platform, value: gcp_kubernetes_engine, action: insert}
      - {key: deployment.environment.name, value: "${env:ENVIRONMENT}", action: upsert}
      - {key: environment, value: "${env:ENVIRONMENT}", action: upsert}

  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
    check_interval: 5s

  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  otlphttp/base14:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}

service:
  pipelines:
    # ...your existing pipelines...
    metrics/nginx:
      receivers: [prometheus/nginx]
      processors: [memory_limiter, resource/nginx, batch]
      exporters: [otlphttp/base14]

The service.name here is deliberately not apigateway-metrics. A managed API Gateway and a self-managed nginx gateway publish different metric vocabularies, and ServiceName is the leading sort key in the Scout data lake — sharing one makes every query read both.

ingress-nginx metrics carry an ingress and host label, giving you the per-route breakdown that neither the nginx receiver nor GCP API Gateway provides.

ingress-nginx path labels

nginx_ingress_controller_requests is labelled by ingress name rather than raw path, so it is bounded. If you enable the optional per-path histogram metrics, they are not — the label takes the request URI verbatim, and any URL containing an id becomes its own series. Normalize the path in the collector before it reaches Scout, or leave those metrics off.

Access logs​

Configure nginx to log JSON so the collector does not have to parse a custom format:

nginx.conf
log_format otel_json escape=json
  '{"time":"$time_iso8601",'
  '"http.request.method":"$request_method",'
  '"url.path":"$uri",'
  '"http.response.status_code":$status,'
  '"http.request.server.duration":$request_time,'
  '"network.peer.address":"$remote_addr",'
  '"user_agent.original":"$http_user_agent",'
  '"server.address":"$upstream_addr",'
  '"trace_id":"$otel_trace_id"}';

access_log /var/log/nginx/access.log otel_json;

Naming the JSON keys after semantic conventions means the filelog receiver's json_parser produces correctly named attributes with no mapping step. $otel_trace_id is available only when nginx-module-otel is loaded, and is what links these logs to traces.

Traces​

nginx-module-otel gives nginx real distributed tracing — it starts or continues a trace at the gateway, so every downstream span sits under a root span that includes the gateway's own time. base14 publishes a prebuilt module at base-14/nginx-otel-build. Configuration is in nginx component.

A self-managed nginx gateway can emit traces; GCP API Gateway cannot.

Correlating with the load balancer​

An nginx gateway on GCP almost always sits behind a Cloud Load Balancer. That means two hops recording the same request, and comparing them is diagnostic:

  • Latency at the LB but not at nginx points at the network between them, or at a backend the LB served without forwarding.
  • Requests at the LB that never appear in the nginx access log were rejected by the LB or Cloud Armor.
  • Status codes that differ between the two hops mean something is rewriting responses.

See Cloud Load Balancing for the LB side. Use the same deployment.environment.name on both so they line up.

Cardinality control​

The two paths fail differently here, which is worth knowing before you enable either.

AttributePathCardinalityKeep?
gateway_id, locationAPI GatewayOne per gatewayYes
api_configAPI GatewayOne per published config revisionOnly during a rollout
response_code_classBoth4Yes
ingress, hostingress-nginxOne per ingress or hostnameYes
Request pathingress-nginx optional metricsUnboundedNo

Managed API Gateway is naturally bounded — it publishes four metrics with a handful of labels, and api_config is the only one that grows, one value per config you publish.

nginx is the opposite. Its optional per-path histograms label by raw request URI, so /orders/8a3f-… and /orders/9b2c-… become separate series forever. Either leave those metrics off, or normalize the path before export:

nginx-gateway-config.yaml
processors:
  transform/nginx:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - replace_pattern(attributes["path"], "/[0-9a-f-]{8,}", "/{id}")
          - replace_pattern(attributes["path"], "/[0-9]+", "/{id}")

Verify​

  1. The collector starts cleanly — check for PermissionDenied (Part A) or scrape errors (Part B) in its logs.

  2. Confirm metrics landed. For API Gateway:

    SELECT MetricName, count() AS points, sum(Value) AS total
    FROM otel_metrics_sum
    WHERE ServiceName = 'apigateway-metrics'
      AND MetricName = 'apigateway.googleapis.com/request_count'
      AND TimeUnix >= now() - INTERVAL 1 HOUR
    GROUP BY MetricName
    SETTINGS max_execution_time = 30, max_rows_to_read = 50000000

  3. For nginx, confirm the scrape target is up. The prometheus receiver emits an up series per target; a value of 0 means the target was discovered but did not respond.

Troubleshooting​

API Gateway metrics are empty although the gateway serves traffic. Confirm project_id is the project holding the gateway, not the one holding the backend. Managed API Gateway and its Cloud Run backend are frequently in different projects.

All API Gateway metrics stopped after adding latency. request_latencies, request_sizes and response_sizes are all distributions and need collector v0.129.0 or later.

ingress-nginx scrape returns nothing. The metrics port is not named metrics in your Helm values, so the relabel rule drops it. Check the container port name on the controller pod rather than assuming the default.

nginx access logs have no trace id. $otel_trace_id is only populated when nginx-module-otel is loaded and tracing is enabled for that location block. Without the module the variable expands to an empty string.

The series count exploded after enabling nginx path metrics. The per-path histograms label by raw request URI. Normalize the path before export or leave those metrics disabled.

Request counts differ between the load balancer and nginx. That is usually correct rather than a collection bug — see Correlating with the load balancer.

FAQ​

Is Google Cloud API Gateway based on nginx?​

No — Google Cloud API Gateway runs ESPv2, which is built on Envoy. If you are looking for nginx telemetry, you are running nginx yourself, and Part B of this guide applies rather than Part A.

How do I monitor GCP API Gateway with OpenTelemetry?​

Use the googlecloudmonitoring receiver against the apigateway.googleapis.com/ prefix for its four metrics, and route resource.type="apigateway.googleapis.com/Gateway" logs through a Log Router sink for per-request detail.

Can GCP API Gateway emit distributed traces?​

GCP API Gateway emits metrics and request logs only, never traces. A self-managed nginx gateway can emit traces using nginx-module-otel, which is one of the stronger arguments for running your own gateway if tracing matters to you.

What is the best way to get metrics from ingress-nginx on GKE?​

Scrape its built-in Prometheus endpoint on port 10254 with the prometheus receiver. It already exposes per-ingress and per-host metrics, so no separate exporter is needed.

Why does the nginx receiver give me so few metrics?​

It reads stub_status, which nginx open source limits to four values. Use nginx-prometheus-exporter or, on Kubernetes, ingress-nginx's own Prometheus endpoint for anything more detailed.

Should I collect metrics at the load balancer or at nginx?​

Collect at both. They see different things, and the difference between them is itself diagnostic — requests the load balancer rejected never reach nginx, and latency added between the two hops shows up nowhere else.

Reference​

Was this page helpful?