Sending Telemetry to Observability Platforms

Like any application instrumented with OpenTelemetry, MetricsHub utilizes the OTLP protocol to transmit data. Although MetricsHub Community can directly send metrics to observability platforms that support OpenTelemetry natively, it is usually recommended in production environments to use an OpenTelemetry Collector to:

• Aggregate metrics across different sources.
• Serve as a proxy, particularly in firewall-secured areas.
• Manage error handling, including retries.

Bundled with OpenTelemetry Collector Contrib, MetricsHub Enterprise facilitates connections to over 30 different observability platforms.

Configure the OTel Collector (Enterprise Edition)

As a regular OpenTelemetry Collector, MetricsHub Enterprise consists of:

• receivers
• processors
• exporters
• and several extensions.

MetricsHub Enterprise 3.9.02 leverages version 0.146.1 of OpenTelemetry.

To configure the OpenTelemetry Collector of MetricsHub Enterprise, edit the otel/otel-config.yaml file.

Locating the Configuration File

Platform	Path
Windows	C:\ProgramData\MetricsHub\otel\otel-config.yaml
Linux	./metricshub/lib/otel/otel-config.yaml

Note: To assist with the setup process, a configuration example of the OpenTelemetry Collector, otel/otel-config-example.yaml, is provided for guidance.

Important: We recommend using an editor supporting the Schemastore to edit MetricsHub's configuration YAML files (Example: Visual Studio Code and vscode.dev, with RedHat's YAML extension).

Receivers

OTLP gRPC

Warning: Only update this section if you customized the MetricsHub Agent settings. The MetricsHub Agent pushes the collected data to the OTLP Receiver via gRPC on port TCP/4317.

The OTLP Receiver is configured by default with the self-signed certificate security/otel.crt and the private key security/otel.key to enable the TLS protocol. If you wish to set your own certificate file, configure the MetricsHub Agent with the correct Trusted Certificates File. Because the OTLP Exporter of the MetricsHub Agent performs hostname verification, you will also have to add the localhost entry (DNS:localhost,IP:127.0.0.1) to the Subject Alternative Name (SAN) extension of the new generated certificate.

Clients requests are authenticated with the Basic Authenticator extension.

otlp:
  protocols:
    grpc:
      endpoint: localhost:4317
      tls:
        cert_file: ../security/otel.crt
        key_file: ../security/otel.key
      auth:
        authenticator: basicauth

OpenTelemetry Collector Internal Exporter for Prometheus

The OpenTelemetry Collector's internal Exporter for Prometheus is an optional source of data. It provides information about the collector activity. It's referred to as prometheus/internal in the pipeline and leverages the standard prometheus receiver.

prometheus/internal:
  config:
    scrape_configs:
      - job_name: otel-collector-internal
        scrape_interval: 60s
        static_configs:
          - targets: [localhost:8888]

Under the service:telemetry:metrics section, you can set the metrics level or the address of the OpenTelemetry Collector Internal Exporter (by default: localhost:8888).

service:
  telemetry:
    logs:
      level: info
    metrics:
      level: basic
      readers:
      - pull:
          exporter:
            prometheus:
              host: 'localhost'
              port: 8888

Processors

By default, the collected metrics go through 5 processors:

• memory_limiter to limit the memory consumed by the OpenTelemetry Collector process (configurable)
• filter to include or exclude metrics
• batch to process data in batches of 10 seconds (configurable).
• metricstransform to enrich the collected metrics, typically with labels required by the observability platforms. The metricstransform processor has many options to add, rename, delete labels and metrics.

Exporters

The exporters section defines the destination of the collected metrics. MetricsHub Enterprise version 3.9.02 includes support for all the OpenTelemetry Collector Contrib exporters, such as:

• Prometheus Remote Write Exporter
• Datadog Exporter
• Debug Exporter
• New Relic (OTLP exporter)
• Prometheus Exporter
• Splunk SignalFx
• and many more...

You can configure several exporters in the same instance of the OpenTelemetry Collector to send the collected metrics to multiple platforms.

Extensions

HealthCheck

The healthcheck extension checks the status of MetricsHub Enterprise . It is activated by default and runs on port 13133 http://localhost:13133.

Refer to Check the collector is up and running for more details.

zpages

The zpages extension provides debug information about all the different components. It notably provides:

• general information about MetricsHub
• details about the active pipeline
• activity details of each receiver and exporter configured in the pipeline.

Refer to Check the pipelines status for more details.

Basic Authenticator

The Basic Authenticator extension authenticates the OTLP Exporter requests by comparing the Authorization header sent by the OTLP Exporter and the credentials provided in the security/.htpasswd file. Refer to the Apache htpasswd documentation to know how to manage user files for basic authentication.

basicauth:
  htpasswd:
    file: ../security/.htpasswd

The .htpasswd file is stored in the security directory.

Warning: If a different password is specified in the .htpasswd file, update the Basic Authentication Header of the MetricsHub Agent.

The Pipeline

Configured extensions, receivers, processors and exporters are taken into account if and only if they are declared in the pipeline:

service:
  telemetry:
    logs:
      level: info # Change to debug for more details
    metrics:
      address: localhost:8888
      level: basic
  extensions: [health_check, basicauth]
  pipelines:
    metrics:
      receivers: [otlp, prometheus/internal]
      processors: [memory_limiter, batch, metricstransform]
      exporters: [prometheusremotewrite/your-server] # List here the platform of your choice


    # Uncomment the section below to enable logging of hardware alerts.
    # logs:
    #   receivers: [otlp]
    #   processors: [memory_limiter, batch]
    #   exporters: [debug] # List here the platform of your choice

Configure the OTLP Exporter (Community Edition)

By default, the MetricsHub Agent pushes the collected metrics to the OTLP Receiver through gRPC on port TCP/4317. To push data to the OTLP receiver of your choice:

• locate the otel section in your configuration file (metricshub.yaml)
• configure the otel.exporter.otlp.metrics.endpoint parameter as follows:

otel:
  otel.exporter.otlp.metrics.endpoint: https://MY_HOST:4317

resourceGroups: #...

where MY_HOST should be replaced with the hostname or IP address of the server where the OTLP receiver is installed.

Additional properties are available, refer to OTLP Exporter settings for details.

Use the below syntax if you wish to push metrics to the Prometheus OTLP Receiver:

otel:
  otel.exporter.otlp.metrics.endpoint: http://PROM_SERVER_HOST:9090/api/v1/otlp/v1/metrics
  otel.exporter.otlp.metrics.protocol: http/protobuf

where PROM_SERVER_HOST should be replaced with the hostname or IP address of the server where Prometheus is running.

Note: For specific configuration details, refer to the OpenTelemetry Auto-Configure documentation. This resource provides information about the properties to be configured depending on your deployment requirements.

Trusted certificates file

If an OTLP Receiver certificate is required, configure the otel.exporter.otlp.metrics.certificate parameter under the otel section:

otel:
  otel.exporter.otlp.metrics.certificate: /opt/metricshub/security/new-server-cert.crt

resourceGroups: # ...

The file should contain one or more X.509 certificates in PEM format.

OTLP Exporter Settings

MetricsHub sends collected metrics to an OTLP Receiver using gRPC or HTTP/Protobuf.

The table below describes the available OTLP Exporter properties:

Property	Description
otel.exporter.otlp.metrics.endpoint	The OTLP metrics endpoint URL. Must be an http or https URL, depending on whether TLS is used. Default: http://localhost:4317 (gRPC) / http://localhost:4318/v1/metrics (HTTP/Protobuf)
otel.exporter.otlp.metrics.protocol	Transport protocol for OTLP metric requests. Possible Values: grpc, http/protobuf. Default: grpc
otel.exporter.otlp.metrics.certificate	Path to a PEM-formatted file containing trusted certificates for verifying the OTLP server’s TLS credentials. Default: Uses the host platform’s trusted root certificates
otel.exporter.otlp.metrics.headers	Custom headers to send with OTLP metric requests, typically for authentication. Default: Not set
otel.exporter.otlp.metrics.timeout	Timeout for OTLP metric requests (in seconds). Default: 10
otel.exporter.otlp.metrics.pool.size	Exporter pool size. This setting directly determines how many metric export operations can run in parallel. Default: 20
otel.exporter.otlp.metrics.append_resource_attributes	When enabled, all resource attributes will be added as attributes on each exported metric. Default: false

Example

To send MetricsHub metrics via gRPC to an OTLP Receiver at https://localhost:4317, including an Authorization header, configure the following in the MetricsHub configuration file (metricshub.yaml):

otel:
  otel.exporter.otlp.metrics.endpoint: https://localhost:4317
  otel.exporter.otlp.metrics.protocol: grpc
  otel.exporter.otlp.metrics.headers: Authorization=<value>

This configuration ensures that metrics are securely transmitted to the specified endpoint.