On This Page

Home / Edge/ Integrations/ Send Data/ Google Cloud/Google Cloud Observability Destination

Google Cloud Observability Destination ​

The Google Cloud Observability Destination enables integration with Google Cloud Observability. This Destination routes observability data, including logs, metrics, and traces, to Google Cloud using the OpenTelemetry Protocol (OTLP).

Type: Streaming | TLS Support: Configurable | PQ Support: Yes

Prerequisites ​

Before you configure this Destination, ensure the following:

Configuring a Google Cloud Observability Destination ​

In Cribl Edge, set up a Google Cloud Observability Destination.

  1. On the top bar, select Products, and then select Cribl Edge. Under Fleets, select a Fleet. Next, you have two options:

    • To configure via QuickConnect, navigate to Routing > QuickConnect (Stream) or Collect (Edge). Select Add Destination and select Google Cloud > Observability, choosing either Select Existing or Add New.
    • To configure via the Routes, select Data > Destinations (Stream) or More > Destinations (Edge). Select Google Cloud > Observability, then select Add Destination to open a New Destination modal.

  2. In the Destination modal, configure the following under General Settings:

    • Output ID: Enter a unique name to identify this output definition. If you clone this Destination, Cribl Edge will add -CLONE to the original Output ID.
    • Endpoint: Read-only. Fixed to telemetry.googleapis.com:443. This is the Telemetry (OTLP) API endpoint for Google Cloud Observability. All three signals share this transport. The OTLP service path determines whether the call lands on traces, metrics, or logs.
    • Backpressure behavior: Optionally, set whether to Block, Drop, or send to Persistent Queue when all receivers are exerting backpressure.
    • Tags: Optionally, add tags that you can use to filter and group Destinations on the Destinations page. These tags aren’t added to processed events. Use a tab or hard return between (arbitrary) tag names.

  3. Optionally, configure any Authentication, TLS, Persistent Queue, Processing, and Advanced settings outlined in the sections below.

  4. Select Save, then Commit & Deploy.

Authentication ​

Use the Google authentication method drop-down to select one of these options:

  • Auto: Uses Google Application Default Credentials (ADC). Cribl Edge uses the GOOGLE_APPLICATION_CREDENTIALS environment variable when it is set to a service account key file or inline credentials. When that variable is unset or invalid, Cribl Edge falls back to other ADC locations described in Google’s documentation. On Google Compute Engine VMs, you can also use credentials attached to the instance.

  • Secret: Select a stored secret that references Google service account credentials. A Create link is available to store a new, reusable secret.

If you authenticate with user credentials through ADC (for example, after running gcloud auth application-default login), add Advanced Settings > Metadata with Key x-goog-user-project and Value set to the Google Cloud project ID that should receive billing and quota attribution.

TLS Settings (Client Side) ​

Use TLS default is toggled off by default. When toggled on:

Validate server certs: Toggle on (default) to reject certificates that are not authorized by a CA in the CA certificate path, or by another trusted CA (for example, the system’s CA).

Minimum TLS version: Optionally, select the minimum TLS version to use when connecting.

Maximum TLS version: Optionally, select the maximum TLS version to use when connecting.

Certificate: Select an existing certificate name from the drop-down or add a new certificate by selecting Create. See Add a TLS Certificate to a Fleet for details.

CA certificate path: Path on client containing CA certificates (in PEM format) to use to verify the server’s cert. Path can reference $ENV_VARS.

Private key path (mutual auth): Path on client containing the private key (in PEM format) to use. Path can reference $ENV_VARS. Use only if mutual auth is required.

Certificate path (mutual auth): Path on client containing certificates in (PEM format) to use. Path can reference $ENV_VARS. Use only if mutual auth is required.

Passphrase: Passphrase to use to decrypt private key.

Persistent Queue Settings ​

The Persistent Queue Settings tab displays when the Backpressure behavior option in General settings is set to Persistent Queue. Persistent queue buffers and preserves incoming events when a downstream Destination has an outage or experiences backpressure.

Before enabling persistent queue, learn more about persistent queue behavior and how to optimize it with your system:

Some of these settings might not appear if you set the persistent queue storage mode to Network filesystem or AWS S3 at the Worker Group level (Worker Group Settings > PQ Storage). When using these storage types, this behavior is managed at the Worker Group level. See Worker Group PQ Storage Fields for more information.

On Cribl-managed Cloud Workers (with an Enterprise plan), this tab exposes only the destructive Clear Persistent Queue button (described at the end of this section). A maximum queue size of 1 GB disk space is automatically allocated per PQ-enabled Destination, per Worker Process. The 1 GB limit is on outbound uncompressed data, and no compression is applied to the queue.

This limit is not configurable. If the queue fills up, Cribl Edge blocks outbound data. To configure the queue size, compression, queue-full fallback behavior, and other options below, use a hybrid Group.

Mode: Use this menu to select when Cribl Edge engages the persistent queue in response to backpressure events from this Destination. The options are:

ModeDescription
ErrorQueues and stores data on a disk when the Destination is unavailable or in an error state.
BackpressureQueues and stores data to a disk when it detects backpressure from the Destination until the backpressure event resolves.
Always OnCribl Edge immediately queues and stores all data on a disk for all events, even when there is no backpressure.

If a Worker/Edge Node starts with an invalid Mode setting, it automatically switches to Error mode. This might happen if the Worker/Edge Node is running a version that does not support other modes (older than 4.9.0), or if it encounters a nonexistent value in YAML configuration files.

File size limit: The maximum data volume to store in each queue file before closing it. Enter a numeral with units of KB, MB, and so on. Defaults to 10 MB. When you save the configuration, File size limit must be greater than or equal to Buffer size limit (bytes). Cribl Edge rejects invalid combinations.

Queue size limit: The maximum amount of disk space that the queue can consume on each Worker Process. When the queue reaches this limit, the Destination stops queueing data and applies the Queue-full behavior. Defaults to 5 GB. This field accepts positive numbers with units of KB, MB, GB, and so on. You can set it as high as 1 TB, unless you’ve configured a different Worker Process PQ size limit on the Worker Group/Fleet Settings page.

Queue file path: The location for the persistent queue files. Defaults to $CRIBL_HOME/state/queues. Cribl Edge appends /<worker-id>/<output-id> to this value.

Compression: Set the codec to use when compressing the persisted data after closing a file. Defaults to None. Gzip is also available.

Queue-full behavior: Whether to block or drop events when the queue begins to exert backpressure. A queue begins to exert backpressure when the disk is low or at full capacity. This setting has two options:

  • Block: The output will refuse to accept new data until the receiver is ready. The system will return block signals back to the sender.
  • Drop new data: Discard all new events until the backpressure event has resolved and the receiver is ready.

Buffer size limit (bytes): The maximum memory to buffer events before flushing them to persistent queue on disk. Enter a value with a unit suffix. For example: 64KB, 1MB, or 10MB. Spaces between the number and unit are allowed (such as 64 KB). Units are case-insensitive. Plain byte values (like 65536) are accepted, but decimals (like 0.5MB) are not. The valid range is 64KB to 10MB. Values outside this range are rounded to the nearest limit. Defaults to 1MB.

Deprecation Notice

The Buffer size limit (bytes) setting replaces the deprecated Max buffer size setting to provide more predictable memory management in version 4.18.0. The Max buffer size setting will be removed in version 4.19.1. For upgraded Worker Groups and Fleets, the new byte-based limit defaults to 1MB. Update your configurations to the new byte-based limit to ensure optimal memory stability.

Strict ordering: Toggle on (default) to enable FIFO (first in, first out) event forwarding, ensuring Cribl Edge sends earlier queued events first when receivers recover. The persistent queue flushes every 10 seconds in this mode. Toggle off to prioritize new events over queued events, configure a custom drain rate for the queue, and display this option:

  • Drain rate limit (EPS): Optionally, set a throttling rate (in events per second) on writing from the queue to receivers. (The default 0 value disables throttling.) Throttling the queue drain rate can boost the throughput of new and active connections by reserving more resources for them. You can further optimize Worker startup connections and CPU load in the Worker Processes settings.

Clear Persistent Queue: For Cloud Enterprise only, select this button if you want to delete the files that are currently queued for delivery to this Destination. If you select this button, a confirmation modal appears. Clearing the queue frees up disk space by permanently deleting the queued data, without delivering it to downstream receivers. This button only appears after you define the Output ID.

Use the Clear Persistent Queue button with caution to avoid data loss. See Steps to Safely Disable and Clear Persistent Queues for more information.

Processing Settings ​

Post-Processing ​

Pipeline: Pipeline or Pack to process data before sending the data out using this output.

System fields: A list of fields to automatically add to events that use this output - both metric events, as dimensions, and log events, as labels. Supports wildcards.

By default, includes cribl_pipe (Cribl Edge Pipeline that processed the event).

Other options include:

  • cribl_host - Cribl Edge Node that processed the event.
  • cribl_input - Cribl Edge Source that processed the event.
  • cribl_output - Cribl Edge Destination that processed the event.
  • cribl_wp - Cribl Edge Worker Process that processed the event.
  • cribl_route - Cribl Edge Route (or QuickConnect) that processed the event.
  • cribl_group - Cribl Edge Fleet of the node that processed the event.
  • cribl_mode - Cribl Edge mode of the node that processed the event.

Advanced Settings ​

Connection timeout: Amount of time (milliseconds) to wait for the connection to establish before retrying. Defaults to 10000.

Request timeout: Amount of time (in seconds) to wait for a request to complete before aborting it. Defaults to 30 sec.

Request concurrency: Maximum number of concurrent requests before blocking. This is set per Worker Process. Defaults to 5.

Keep alive time (seconds): How often the sender should ping the peer to keep the connection alive. Defaults to 30.

Body size limit (KB): Maximum size of the request body sent to Google Cloud Observability. Defaults to 2048 KB. The minimum is 1024 KB and the maximum is 10240 KB. The actual request body size might exceed the specified value because the Destination adds bytes when it writes to the downstream receiver. Cribl recommends that you experiment with the Body size limit value until downstream receivers reliably accept all events.

Events-per-request limit: Maximum number of events to include in each gRPC request. Defaults to 0 (unlimited). Use this setting to stay within Google Cloud Observability request limits. For metrics, combine it with the OTLP Metrics function Batch size setting.

Flush period (sec): Maximum time between requests. Low values could cause the payload size to be smaller than the configured Body size limit. Defaults to 1 sec.

Metadata: Extra information to send with each gRPC request. Select Add Metadata to add each item as a Key-Value pair. The Key field is arbitrary. The Value field is a JavaScript expression that is evaluated just once, when this Destination is initialized. If you pass credentials as metadata, Cribl recommends using C.Secret().

Use dynamic metadata: Toggle on to enable per-event dynamic metadata on outbound gRPC requests. When enabled, Cribl Edge reads metadata key-value pairs from the field specified in Dynamic metadata field on each event. It injects them into the gRPC request metadata and batches together events that share the same metadata values. Dynamic metadata entries override any static Metadata entries with matching keys. Default is off.

Dynamic metadata field: The event field that contains the metadata key-value pairs to inject into outbound gRPC request metadata. Visible only when Use dynamic metadata is enabled. Defaults to __headersOut.

Environment: Optionally, specify a single Git branch on which to enable this configuration. If this field is empty, the config will be enabled everywhere.

Protocol and Transport Support ​

This Destination always sends observability data using the OpenTelemetry Protocol (OTLP) v1.3.1 over gRPC to telemetry.googleapis.com:443. These transport settings are fixed and are not shown in the configuration UI. OTLP defines Protocol buffer (Protobuf) schemas for its payloads (requests and responses).

When configuring Pipelines (including pre-processing and post-processing Pipelines), you need to ensure that events sent to this Destination conform to the relevant Protobuf specification.

This Destination will drop non-conforming events. If the Destination encounters a parsing error when trying to convert an event to OTLP, it discards the event and Cribl Edge logs the error.

For metric naming and label constraints enforced by Google Cloud Observability, see OTLP metric ingestion overview.

OTLP Resource Attributes ​

Google Cloud Observability maps OTLP resource attributes to monitored resource types and labels for logs, metrics, and traces. The required attributes depend on the signal type and the source environment (for example, Compute Engine, GKE, or on-premises hosts).

Use Pipelines to add or transform resource attributes before events reach this Destination. For workloads running in Google Cloud, you can also attach resource attributes at the Collector or SDK layer using a Google Cloud resource detector.

At minimum, ensure that telemetry includes a project identifier such as gcp.project_id or project_id so Google Cloud Observability can route data to the correct project. See Google’s Write OTLP logs to the Telemetry API topic for examples.

Google Cloud Roles and Permissions ​

The Google identity used by this Destination needs permission to write telemetry to your Google Cloud project. Grant at least the Cloud Telemetry Writer role (roles/telemetry.writer), which includes:

  • logging.logEntries.create
  • monitoring.timeSeries.create
  • telemetry.traces.write

Enable the Telemetry API (telemetry.googleapis.com), Cloud Logging API (logging.googleapis.com), Cloud Monitoring API (monitoring.googleapis.com), and Cloud Trace API (cloudtrace.googleapis.com) in the target project.

Troubleshooting ​

The Destination’s configuration modal has helpful tabs for troubleshooting:

Live Data: Try capturing live data to see real-time events as they flow through the Destination. On the Live Data tab, click Start Capture to begin viewing real-time data.

Logs: Review and search the logs that provide detailed information about the delivery process, including any errors or warnings that may have occurred.

Test: Ensures that the Destination is correctly set up and reachable. Verify that sample events are sent correctly by clicking Run Test.

You can also view the Monitoring page that provides a comprehensive overview of data volume and rate, helping you identify delivery issues. Analyze the graphs showing events and bytes in/out over time.