OTLP / OpenTelemetry

SenHub Agent can push metrics, logs and traces natively over OTLP/gRPC or OTLP/HTTP to any OpenTelemetry receiver — an OTel collector, vmagent's OTLP endpoint, a direct VictoriaMetrics / VictoriaLogs ingest, Grafana Cloud OTLP, etc.

OTLP is a push path, complementary to the Prometheus pull endpoint. Both signals carry the same metric names, types, units, and attributes because both are produced by the same neutral OTel mapper internally — operators querying Prometheus today and switching to OTLP push tomorrow do not rewrite a single PromQL query.

Quick start

The fastest path is to provision OTLP at install time — one flag writes a ready-to-use strategies.d/10-otlp.yaml:

senhub-agent config init --otlp-endpoint otel-collector.internal:4317
# direct to a native VictoriaMetrics / Grafana Alloy OTLP/HTTP endpoint:
senhub-agent config init --otlp-endpoint vm.internal:4318 --otlp-protocol http

This enables metrics + logs export out of the box. To wire it by hand instead:

1. Create strategies.d/10-otlp.yaml:

otlp:
  endpoint: "otel-collector.internal:4317"   # required
  tls:
    enabled: true
  signals:
    metrics:
      enabled: true
      interval: 30s
    logs:
      enabled: true

An installation that still uses the legacy monolithic agent-config.yaml holds the same parameters under storage:, in the params of an entry named otlp.

1. Save the file: the running agent picks the output up without a restart. It connects lazily, so a collector that is not reachable yet does not block anything; failures appear as retried export attempts.

2. Verify the collector is receiving data:

# At the collector side
journalctl -u otel-collector -f
# or query VictoriaMetrics directly
curl -G http://vm:8428/api/v1/query \
  --data-urlencode 'query=count by(__name__) ({service_name="senhub-agent"})'

Configuration

The full set of knobs (every key maps directly to an OTel SDK option):

# strategies.d/10-otlp.yaml
otlp:
  # Required: host:port of the receiver, no scheme prefix.
  endpoint: "otel-collector.internal:4317"

  # Transport: grpc (OTLP/gRPC, port 4317) or http (OTLP/HTTP protobuf,
  # port 4318). "http/protobuf" is accepted as an alias of http.
  protocol: grpc                  # default grpc

  # Standby receivers, tried in order when an export to the primary
  # fails; the agent returns to the primary once it recovers.
  fallback_endpoints:
    - "otel-collector-standby.internal:4317"

  # Optional multi-tenant routing: sugar for the X-Scope-OrgID header
  # (the standard tenant key for Mimir / Loki / Tempo / VictoriaMetrics).
  # "org_id" is accepted as an alias. Env-expandable.
  tenant: "acme"

  # Optional headers (for example bearer auth at the gateway).
  headers:
    Authorization: "Bearer YOUR-INGEST-TOKEN"

  # TLS — defaults to enabled. Disable explicitly for plaintext
  # localhost / lab environments.
  tls:
    enabled: true                 # default true
    insecure_skip_verify: false   # opt-in for self-signed certs in test
    ca_file: /etc/ssl/private/otlp-ca.pem
    cert_file: /etc/ssl/private/agent.pem    # mTLS, optional
    key_file:  /etc/ssl/private/agent.key    # required if cert_file set

  # Base path for backends that serve OTLP under a prefix rather than
  # at the root (see below). OTLP/HTTP only.
  url_path_prefix: "/api/v2/otlp"

  # Close an idle HTTP connection before the ingress does (see below).
  # OTLP/HTTP only; unset keeps the Go default of 90s.
  idle_conn_timeout: 45s

  compression: gzip               # gzip | none — default gzip
  timeout: 60s                    # per-export deadline — default 60s

  retry:
    enabled: true
    initial_interval: 5s
    max_interval: 30s
    max_elapsed_time: 1m

  signals:
    metrics:
      enabled: true
      interval: 30s               # push cadence
      temporality: cumulative     # cumulative | delta — default cumulative
    logs:
      enabled: true
      batch_size: 1000            # max records per gRPC export
      batch_timeout: 5s           # flush even if batch_size not reached
      buffer_size: 10000          # bounded queue; drop-oldest beyond
    traces:
      enabled: true               # default false — relays spans ingested by an
                                  # otlp_receiver probe (signals: [traces])
      batch_size: 512
      batch_timeout: 5s
      buffer_size: 2048           # bounded queue; drop beyond
      sample_ratio: 1.0           # head sampling, 0.0-1.0

  # Telemetry forwarded on behalf of applications (otlp_receiver).
  relay:
    enrichment: true              # default true; add the agent's context
                                  # to relayed telemetry (see below)

  # Cardinality caps and staleness (see "Cardinality controls").
  max_store_size: 50000           # default 50000 series in all; 0 = unbounded
  max_active_series_per_probe: 10000  # default 10000 per probe; 0 = no per-probe cap
  staleness_ttl: 10m              # default 10m; a series silent this long is
                                  # evicted. 0s disables eviction

  # How many exports may be in flight at once. The push splits a
  # large cycle per probe and ships the parts in parallel; 1 means
  # the single-batch path. Accepted range 1-64.
  max_concurrent_exports: 4       # default 4

  # Cap the memory the metric store may use, in MiB of Go heap.
  # Past the soft limit the store keeps its existing series and
  # refuses new ones; past the hard limit it refuses all writes and
  # forces a collection. 0 disables either threshold.
  memory_limit:
    soft_mib: 200                 # default 200
    hard_mib: 400                 # default 400
    check_interval: 5s            # default 5s

  # Survive a restart: the last value of every series is written to
  # disk and restored at boot, so cumulative counters continue
  # instead of resetting.
  persistence:
    enabled: true                 # default true: the logs queue is on in the
                                  # state directory; the checkpoint stays off
                                  # until `path` is set; enabled: false turns
                                  # both off
    path: /var/lib/senhub-agent/otlp   # checkpoint and logs queue directory;
                                  # empty = no checkpoint, queue in the state
                                  # directory (see "Logs survive an outage")
    interval: 30s                 # default 30s
    # Disk cap for the logs dead-letter queue, which holds batches
    # the receiver could not take during an outage. Past it the
    # oldest batches are evicted. 0 keeps the default.
    logs_queue_max_bytes: 134217728   # default 128 MiB
    # Batches that waited longer than this are dropped, at boot and
    # every 10 minutes. 0 = no age limit.
    logs_queue_max_age: 24h       # default 24h

  # Resource attributes attached to every emitted batch. Defaults
  # are derived from agent identity if omitted.
  resource:
    service.name: senhub-agent             # default
    service.instance.id: <uuid>             # default: an RFC 4122 UUID derived from agent.key
    deployment.environment: prod
    # Any additional keys are passed through as resource attributes.
    k8s.cluster.name: edge-01

tenant (optional multi-tenant routing)

Sets the X-Scope-OrgID header — the standard tenant key used by Mimir, Loki, Tempo and VictoriaMetrics — on every signal. org_id is accepted as an alias; an explicit X-Scope-OrgID in headers: takes precedence. The value is env-expandable (tenant: "${env:ORG_ID}").

Ingest authentication is enforced at the collector/edge (a standard OTLP authenticator: bearer token, basic auth, OIDC or mTLS), not by the agent. A self-hosted edge trusts the agent's tenant; an untrusted multi-tenant edge re-derives it from the authenticated token. Your agent license does not gate sending data — it only governs which paid probes may collect; an unlicensed agent can still push over OTLP.

endpoint (required)

host:port of the OTLP receiver, without a scheme prefix, for both transports. protocol selects the transport: grpc (the default, OTLP/gRPC, usual port 4317) or http (OTLP/HTTP protobuf, usual port 4318). With http, the agent builds the URL itself: https:// when tls.enabled is true, http:// otherwise, then the standard /v1/metrics, /v1/logs and /v1/traces paths. There is intentionally no default for this field — silently shipping data to localhost when the operator forgets to set it would be a worse failure mode than refusing to start.

A standby ingress can be listed under fallback_endpoints: the agent prefers the primary, switches on a failed export and returns on its own once the primary recovers. See the backpressure guide for the shape and the trade-offs.

url_path_prefix (OTLP/HTTP only)

endpoint is a host:port pair, so it cannot carry a path. Backends that serve OTLP under a base path instead of at the root need this prefix, which the agent prepends to the standard signal paths: /api/v2/otlp yields /api/v2/otlp/v1/metrics, /api/v2/otlp/v1/logs and /api/v2/otlp/v1/traces.

Setting it with protocol: grpc is refused at config load rather than silently ignored: the gRPC transport addresses services, not URL paths.

Pushing straight to Dynatrace, which serves OTLP at /api/v2/otlp, expects delta temporality and authenticates with an API token:

otlp:
  protocol: http
  endpoint: "abc12345.live.dynatrace.com:443"
  url_path_prefix: "/api/v2/otlp"
  tls:
    enabled: true
  headers:
    Authorization: "Api-Token ${env:DT_API_TOKEN}"
  signals:
    metrics:
      enabled: true
      temporality: delta
    logs:
      enabled: true

idle_conn_timeout (OTLP/HTTP only)

Load balancers and reverse proxies close connections that have been idle for some time. A signal that pushes continuously never reaches that point, but a sparse one does: the agent then discovers the connection is gone only when it tries to use it, and pays a failed request before reconnecting. Logs are the signal this affects, since metrics push on a fixed interval and keep their connection warm.

Setting idle_conn_timeout below your ingress idle timeout makes the agent close first, turning that failure into a clean reconnect. Unset, the Go default of 90 seconds applies, which is longer than most ingress timeouts.

The cost is one extra TCP and TLS handshake per idle period, on a signal that by definition is not busy. Setting it with protocol: grpc is refused at config load: gRPC connection keepalive is a separate mechanism.

tls

Default is enabled: true. For production push to a remote collector you should always keep TLS enabled. The plaintext path is provided only for localhost / lab tests where the gRPC traffic never leaves the loopback interface.

mTLS is supported by setting both cert_file and key_file. Setting only one is rejected at config validation.

signals.metrics and signals.logs

Each signal can be enabled/disabled independently. Disabling one stops the corresponding pipeline and keeps the gRPC connection idle.

temporality defaults to cumulative (the OTel default since OTel v1.0). Cumulative means counter values are reported as absolute totals since the agent start; consumers that prefer deltas (some Datadog setups, vmagent ingest) can set temporality: delta.

signals.traces

The agent produces no spans of its own. The traces signal relays spans received by an otlp_receiver probe configured with signals: [traces]; without such a probe there is nothing to export and the signal stays idle.

Correlation enrichment (relay.enrichment, default true). Relayed spans keep the emitting application's own identity (service.name, service.instance.id, host.*) — the agent never overwrites it. On top of that, the agent inserts its own tenancy context so you can pivot from an app trace to the infrastructure telemetry of the same tenant: your global_tags (for example tenant, site) and deployment.environment are added only if the application didn't already set that key. Only standard / operator-defined attributes are used — no product-specific keys.

Set relay.enrichment: false for a verbatim pass-through. The older spelling signals.traces.relay_enrichment is still read, so a host that set it keeps its behaviour, but relay.enrichment is the key to write: it covers every relayed signal, not traces alone. When a single agent relays traffic for several clients (a shared gateway), assign each source its own tenant with per-source rules:

        traces:
          enabled: true
          relay_tenant_overrides:
            - match: { key: "service.namespace", value: "client-b" }
              tags: { tenant: "client-b", site: "lyon" }

A rule applies to relayed spans whose resource attribute key equals value; its tags are inserted (only when absent) instead of the default global_tags.

Note

service.instance.id is not a join key between the agent's own metrics/logs and a relayed third-party trace — they are different services. The reliable cross-signal pivot is tenant/site, i.e. "this app trace is slow → show the infra telemetry of the same tenant".

signals.entities

Entity events (the infrastructure graph: hosts, interfaces, services and their relationships) ride the OTLP log signal, so this signal has no endpoint or batch knobs of its own — it reuses the log transport.

When entities.enabled is off, the output logs one warning at start, entity emission is off on this OTLP output, so an empty topology can be traced to its cause from the agent log.

    entities:
      enabled: true            # opt-in (default false)
      interval: 60s            # heartbeat cadence; also the liveness backstop
      buffer_size: 256         # bounded queue; drop-oldest beyond
      depends_on_enabled: false  # opt-in (default false): outbound dependency
                                 # edges. Needs root — see below
      depends_on_debounce: 3   # scrapes (not necessarily consecutive) before an outbound
                               # dependency edge is emitted (>= 1, default 3)
      depends_on_exclude_cidrs: []  # peer ranges dropped before anything is
                                    # emitted (operator privacy filter)
      redact_attributes: []    # attribute keys DROPPED from every entity
                               # event before export (default: none)
      governance:              # operator-asserted ownership / criticality /
        criticality: low       # location / lifecycle, stamped on the entities
        lifecycle: active      # this agent emits (default: none)
        owner:
          team: sre
          contact: sre@example.com
        location:
          site: paris
          rack: R12
        labels:
          environment: staging

governance stamps operator-asserted facts on the entities this agent emits — who owns them, how critical they are, where they physically sit, and where they are in their lifecycle. The agent cannot discover any of it; it comes from you. A topology consumer can then filter or group on those keys, so an incident view can be narrowed to one team's critical estate without touching the agent configuration on every host.

criticality takes critical / high / medium / low, and lifecycle takes active / maintenance / decommissioning / retired. labels is a free key/value map for anything that is yours alone.

This block describes the host, and descends from it, on two rules that differ because the two halves of the vocabulary answer different questions.

The location (site, datacenter, rack, room) says where a thing physically is. What runs on this host is where the host is, so it descends to every entity the agent places here: a database on the loopback address, a container, a listening port.

Ownership (owner, criticality, lifecycle) says who answers for a thing. It descends to the entities an operator reasons about, a service instance, a container, a database, a network device, and not to a listening port, which reaches its host in one runs_on hop: the graph already answers "who owns this port" by traversal rather than by copying the fact onto several hundred entities. Criticality descends knowing it errs high, since a backup daemon on a critical host is not itself critical; claiming too much is safer than claiming too little, and a probe that knows better overrides it.

labels do not descend, and the head of an application chain is why. "This host is part of application X" is not "everything running on this host is application X". That holds on a dedicated machine and fails on a shared one, and a cluster node is the shared case: on a host carrying one application label, thirteen entities belonging to five different sets would have inherited it, so a filter that returns exactly the chain would have returned a mixture. Labels stay at host grain. Put one on a probe entry when you want it on what that probe reports.

Neither descends to anything remote. A probe reading a database on another machine reports an entity that does not run here, and giving it this host's owner would be wrong rather than merely noisy. What a probe observes is governed on the probe entry instead, with the same vocabulary: see Governance per probe. A value the probe sets itself always wins over what it would inherit.

These attributes are the producer's facts, in the frozen entity.* and semantic-convention vocabulary. An annotation a consumer adds beside them is a comment, never a correction: once the agent asserts an owner, the way to change it is this configuration.

These keys are a fallback

The depends_on_* keys and interval below are read only when the configuration has no top-level entities: block. When that block exists, even with enabled: false, it wins and these keys are ignored. See Entities Section.

depends_on_enabled turns on outbound dependency discovery — the edges that say "this service talks to that endpoint". It is off by default because mapping a host's connections can be privacy-sensitive.

Outbound dependency discovery needs root

Mapping a socket to the process that owns it reads /proc/<pid>/fd, and only the owner may read it. The agent runs as a service account by default, so it can attribute its own connections and nothing else — every other service's dependencies are invisible.

On a non-root install the agent says so once at startup, naming the counts and the account, and reports its view as failed rather than reporting an empty graph as fact. Leave the option off, or run the agent as root if you want the whole host's dependencies.

depends_on_debounce controls how durable an outbound connection must be before it appears as a depends_on edge: a peer endpoint must be seen on this many emission scrapes before its edge is emitted, which keeps a single stray socket out of the graph. The scrapes need not run consecutively: a peer that is seen, missed, then seen again keeps its hits. A peer keeps its progress across up to fifteen scrapes without being seen, because a short-lived flow is precisely one that is missing from most samples: a reverse proxy that opens a request to a backend and closes it may appear in four samples out of fifteen and never in two running ones, and it is still a real dependency. The soonest a dependency can surface is depends_on_debounce x interval (so the default 3 x 60s is about three minutes); lower it for a more responsive graph, raise it to demand more evidence.

The same window applies once the edge exists: an edge survives up to fifteen consecutive scrapes without the peer being seen (or depends_on_debounce, if larger) before it is given up, and then has to earn its way back. A long-lived connection the socket table happens to miss once is not a dependency that ended, and retracting it on a single miss would reach a topology consumer as an edge flapping in and out.

Loopback peers on an OS dynamic port, the client side of a one-shot local exchange, are ignored whatever these settings say (since 0.6.2).

redact_attributes lists descriptive attribute keys the agent removes from every entity event before export — useful when the entity stream transits a shared collector or a third-party OTLP pipeline that should not see inventory identifiers such as hardware serials or cloud account ids:

      redact_attributes: [hw.serial_number, cloud.account.id]

The listed keys are dropped, not masked: they simply never appear on the wire (an entity state is a full snapshot, so absence is the clean form of redaction — a *** placeholder would pollute the graph with a fake shared value). The filter applies to every entity this strategy emits — the host entity and probe-emitted entities alike — and is per-strategy: a second otlp strategy pointed at a trusted backend can keep the full attribute set while this one redacts.

Two rules to know:

• Identity keys cannot be redacted. Keys that identify an entity (host.id, service.instance.id, container.id, network.device.id, db.instance.id, vmid) are refused at config load with an error — removing an entity's identity would destroy the entity, not protect it. Any other key is accepted.
• Trade-off on hw.serial_number. The hardware serial is the evidence an entity backend uses to reconcile the in-band host entity with an out-of-band BMC (Redfish) view of the same machine. Redacting it disables that same_as reconciliation for this export — list it only when the receiving backend must not see serials.

resource

Resource attributes are attached once per batch by the SDK (per-batch in OTLP wire format) and are how observability backends group series and logs into entities. Defaults:

• service.name = "senhub-agent"
• service.instance.id = an RFC 4122 UUID (version 5) derived from the agent key: one per agent, stable for as long as the key is, as the OpenTelemetry semantic conventions recommend
• service.version = the agent build version (when known)

Any additional key/value pair under resource: is passed through verbatim — useful for k8s.*, cloud.*, host.*, or your own custom labels.

Naming alignment with Prometheus

Metrics emitted via OTLP carry the same internal names as the Prometheus exposition. The path is:

probe data → otelmapper.Resolve → []OtelRecord → { Prometheus serializer
                                                  OTLP exporter }

Both consumers read from the same OtelRecord stream. Prometheus applies its sink-specific conventions (senhub_ prefix, _seconds suffix, dots → underscores), and OTLP keeps the dotted OTel names intact. The OTel collector / VictoriaMetrics / Grafana that ingests the OTLP push automatically applies the same transforms when storing into a Prometheus-compatible TSDB, so the observed metric names are identical at the query layer.

OTel name (OTLP wire)	Prometheus exposition	After OTLP ingest in VictoriaMetrics
system.cpu.utilization (gauge, 1, cpu.mode=user)	senhub_system_cpu_utilization_ratio	system_cpu_utilization_ratio
system.memory.usage (updowncounter, By)	senhub_system_memory_usage_bytes	system_memory_usage_bytes
system.network.io (gauge, By/s, network.io.direction=receive)	senhub_system_network_io_bytes_per_second	system_network_io_bytes_per_second
senhub.netscaler.lbvserver.connections.active	senhub_netscaler_lbvserver_connections_active	senhub_netscaler_lbvserver_connections_active

The collector applies the standard senhub_ prefix loss when going through prometheusremotewrite, but the dimensions (probe_name, probe_type, semantic-convention attributes like cpu.mode, system.memory.state, hw.state) flow through identically. Aliasing both paths under one PromQL query is straightforward.

Logs signal

The OTLP strategy ships logs from these probe sources:

• syslog probe — every received syslog message becomes an OTel log record with severity mapped per the standard RFC 5424 → OTel table (info → INFO, error → ERROR, emergency → FATAL4, etc.) and attributes syslog.facility, syslog.hostname, syslog.appname, …
• event probe — every accepted HTTP event becomes a log record with severity from the event's severity field (EMERG..DEBUG) and the payload fields under the senhub.event.* namespace.
• linux_logs probe — reads the local systemd journal via journalctl --output=json --follow and emits a log per entry with host.name, systemd.unit, syslog.appname, process.pid, … See linux_logs probe.
• filetail probe — each line tailed from a watched log file becomes a log record.
• windows_eventlog probe — each entry read from a subscribed Windows Event Log channel becomes a log record.
• snmp_trap probe — each received SNMP trap becomes a log record.
• otlp_receiver probe with signals: [logs] — relays log records ingested over OTLP from other emitters.

Probes still emit DataPoints to the existing PRTG/Nagios/event strategies; the logs path is additive.

Logs survive an outage

When the collector cannot take a batch of event logs, the agent keeps it on disk and sends it again. A batch is deleted from disk only after the collector has acknowledged it, so a failed or interrupted attempt never removes anything. This is on by default on every install; nothing to set.

While the collector is known to be down, new batches go straight to disk and the agent does not try the network for each one. It probes the collector on a retry clock of 5, 10, 20, 40, 80 and 160 seconds, then every 5 minutes; the clock starts again at 5 seconds after the next outage. As soon as one export succeeds it sends the backlog, oldest first, stopping at the first batch the collector does not acknowledge.

Situation	Event logs	Entity events	Metrics
Collector down, agent running	Kept on disk, sent in order once the collector answers	Re-sent at the next heartbeat	Last value of each series kept in memory
Agent stopped normally (restart, upgrade) during an outage	The records still in memory are written to disk in the first second of the stop, without trying a collector that is down, then sent at the next boot with the rest	Re-sent at the next heartbeat	Lost unless persistence.path is set
Agent restarted, collector still down	Kept on disk, probed on the retry clock	Re-sent at the next heartbeat	Lost unless persistence.path is set
Agent killed or host crashed (kill -9, power cut)	Batches already on disk are kept; records not yet on disk are lost: at most the batch interval (5 s by default) plus one export timeout (4 s), so under 10 s of logs	Re-sent	Same as above
Outage longer than 24 hours or past 128 MiB	The oldest batches are dropped and counted	Re-sent	Same as above

Delivery is at-least-once: if the collector received a batch but its acknowledgement was lost, the batch is sent again, so a record can arrive twice. Records replayed after an outage can also arrive after newer ones; they keep their original timestamp, so order by the record timestamp and deduplicate on timestamp, body and attributes if duplicates matter.

Where the queue lives:

Install	Directory
Linux package (systemd)	/var/lib/senhub-agent/otlp-queue (STATE_DIRECTORY)
Windows (MSI)	C:\ProgramData\SenHub\otlp-queue
Container	$SENHUB_STATE_DIR/otlp-queue (default /var/lib/senhub-agent/otlp-queue)
persistence.path set	<path>/logqueue

The queue is bounded two ways: logs_queue_max_bytes (128 MiB, oldest batches evicted first) and logs_queue_max_age (24 hours). Each sweep that drops aged batches logs one warning with the count, and the records are counted in senhub.agent.otlp.dropped with reason="dropped_by_age" (logs_queue_full for the size cap). During an outage the queue can use up to 128 MiB of the state directory. If the directory is not writable the agent logs one warning and runs without the queue.

Environment variables, which win over the file:

Variable	Default	Effect
SENHUB_LOG_QUEUE	true	false turns the queue off
SENHUB_LOG_QUEUE_RETENTION	24h	Maximum age of a queued batch, a duration such as 6h; 0 = no limit
SENHUB_LOG_QUEUE_MAX_BYTES	134217728	Disk cap, in bytes or with a suffix (64MiB)

In a container the state directory must be on a persistent volume (a mounted volume, or an Azure Files share on Container Apps); without it the queue disappears with the container and only survives an outage, not a restart.

Severity mapping

Source value	OTel SeverityNumber	OTel SeverityText
Syslog 0 — emergency	24	FATAL4
Syslog 1 — alert	23	FATAL3
Syslog 2 — critical	22	FATAL2
Syslog 3 — error	17	ERROR
Syslog 4 — warning	13	WARN
Syslog 5 — notice	10	INFO2
Syslog 6 — info	9	INFO
Syslog 7 — debug	5	DEBUG

Same table is used by linux_logs (PRIORITY field of the journal entry) and the event probe (when the operator-supplied severity string maps to one of the eight standard names).

Routing probes to OTLP

Every metric probe routes to OTLP automatically when an otlp storage is configured — the strategy is included in the default target list alongside senhub, prtg, http. There's nothing to wire per probe.

The data_store router silently skips strategies that aren't configured, so probes that target [senhub, prtg, http, otlp] continue to work in agents where the operator has not enabled OTLP.

Receiver setups

OpenTelemetry Collector (recommended)

A minimal collector that accepts OTLP/gRPC and forwards to VictoriaMetrics + VictoriaLogs:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"

exporters:
  prometheusremotewrite:
    endpoint: "http://vm.internal:8428/api/v1/write"
    resource_to_telemetry_conversion:
      enabled: true

  otlphttp/logs:
    logs_endpoint: "http://victorialogs.internal:9428/insert/opentelemetry/v1/logs"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      exporters: [otlphttp/logs]

resource_to_telemetry_conversion: enabled: true ensures the OTel resource attributes (service.name, service.instance.id, etc.) become Prometheus labels rather than being discarded — usually what operators want.

Single public intake for a fleet (collector-fronted)

The recommended pattern for a fleet of agents across sites: every agent — local or remote — pushes its whole OTLP stream (metrics, logs, entities) to one public HTTPS endpoint, fronted by a reverse proxy that terminates TLS and forwards to a central collector. The collector authenticates the bearer token and fans out to the storage backends.

Agent-side, the entire client configuration is one strategy fragment:

# strategies.d/10-otlp.yaml
otlp:
  endpoint: "ingest.example.com:443"
  tls:
    enabled: true
  headers:
    Authorization: "Bearer ${env:OTLP_BEARER_TOKEN}"
  compression: gzip
  timeout: 60s
  signals:
    metrics:
      enabled: true
      interval: 30s
    logs:
      enabled: true
    entities:
      enabled: true
      interval: 5m
  resource:
    service.name: my-host-prod        # one value per agent
    deployment.environment: prod

Notes:

• One URL, one token, four signals (metrics, logs, traces, entities). Entity events ride the logs transport — enabling signals.entities requires no extra endpoint.
• The bearer token comes from the environment (${env:}) or a root-only file (${file:/etc/senhub-agent/bearer.token}); never inline it.
• host.id, host.name and os.* resource attributes are auto-detected and attached to every signal — don't set them manually.
• host.id is the machine's own identifier: on Linux the SMBIOS product UUID when it is readable, otherwise /etc/machine-id; on Windows the MachineGuid registry value. The Windows value is not the SMBIOS UUID a hypervisor or a BMC reports for the same machine, so join a Windows host to its hardware on host.name, not on host.id.
• The resource: service.name override groups telemetry only. The agent's own service.instance entity always carries service.name: senhub-agent, whatever the override says, so a fleet inventory filtered on that name sees every agent. The per-host label stays available on the telemetry resource and on the host entity.
• gRPC works through standard reverse proxies. nginx needs one location for OTLP/gRPC and one for OTLP/HTTP (the agent uses the standard /v1/* paths):

server {
    listen 443 ssl;
    server_name ingest.example.com;

    # OTLP/gRPC -> collector
    location /opentelemetry.proto.collector. {
        grpc_pass grpcs://127.0.0.1:4317;
    }
    # OTLP/HTTP -> collector
    location /v1/ {
        proxy_pass http://127.0.0.1:4318;
    }
}

Authentication belongs on the collector (for example the bearertokenauth extension), keeping the proxy a pure transport. To feed a second consumer (another OTLP backend, an entity-graph store, ...), fan out at the collector — add a second exporter there — rather than configuring a second OTLP strategy on every agent.

vmagent's native OTLP endpoint

vmagent v1.95+ ingests OTLP/gRPC directly. Configure the agent to push at vmagent.internal:4317 — no collector needed.

Grafana Cloud OTLP

Grafana Cloud accepts OTLP push at a public gRPC endpoint. Get the endpoint URL and ingest token from the Grafana Cloud console:

otlp:
  endpoint: "otlp-gateway-prod-eu-west-2.grafana.net:443"
  headers:
    Authorization: "Basic eW91...="   # from Grafana Cloud console
  tls:
    enabled: true

Agent self-observability

The agent's existing senhub_agent_* metrics (uptime, probe counts, collect errors, HTTP requests, build info) are emitted on the Prometheus path only — the OTLP path does not push agent self- metrics, to avoid feedback loops where a failing OTLP export would itself emit metrics that need to go through OTLP.

The collector's own self-metrics (otelcol_exporter_sent_*, otelcol_receiver_accepted_*) cover the OTLP transport health from the receiver side.

Monitoring the OTLP pipeline

The agent maintains its own snapshot of OTLP pipeline counters that you can read without scraping Prometheus, via three surfaces:

• CLI — senhub-agent status --otlp appends a four-section block (Pipeline / Store & Export / Checkpoint / Parallel) to the standard status view. See the CLI reference.
• Web console — the OTLP output page shows the same counters in its right column, next to the connection test. See Web console.
• JSON endpoint — GET /api/{agentkey}/info/otlp returns the same data as a single JSON snapshot. Useful for custom dashboards or external alerting. The field reference lives in OTLP observability (admin guide).

What to look at

Symptom	What to check
Operator dashboards lag the agent by minutes	Export last and Export mean durations — sustained values close to the configured timeout (default 60 s) mean the sink is the bottleneck.
Series are missing without an explicit error	Dropped total and its by_reason breakdown. probe_cardinality = the per-probe budget was hit; store_cap = the global cap; memory_soft_limit/memory_hard_limit = the agent shed load to stay under its memory budget.
Recovery after a restart looks slow	Checkpoint.Restored entries — if 0 with persistence enabled, the checkpoint wasn't found or failed to parse (check Errors by stage).
Throughput plateaus despite headroom on the collector	Parallel.Sub-batches — value of 1 means the single-batch path; > 1 means the per-probe fan-out is firing.

Cardinality controls

If Dropped by probe_cardinality keeps climbing, lift the per-probe budget in your strategy YAML:

otlp:
  endpoint: "otel-collector.internal:4317"
  max_active_series_per_probe: 20000   # default 10000
  max_store_size: 100000               # default 50000 (global cap)
  staleness_ttl: "10m"                 # default 10m: a series with no new
                                       # datapoint for this long is evicted
                                       # instead of re-exporting forever
                                       # (zombie series); "0s" disables

Both knobs are gauges of last resort — they protect the agent's memory under runaway-cardinality conditions. Raise them only when you know what's generating the series, otherwise the agent will happily eat RAM tracking churning timeseries.

What time a point carries

Every push re-exports the last value of every stored series, so the time stamped on a point decides what a consumer sees between two runs of a probe:

• While the probe still vouches for the value, the point carries the export time. A probe vouches for its last value for one and a half times its interval, plus one push interval. A gauge from a probe that runs every 30 minutes is therefore published as current every 30 seconds, and an alert with a five-minute lookback keeps seeing it.
• Once that window has elapsed, the point carries the time the value was measured: the probe missed its own next run, or the target it measured is no longer in its configuration, and re-stamping the old value would assert a measurement that never happened. The series is evicted after staleness_ttl.
• A series restored from the checkpoint at start-up carries its measurement time until its probe observes it again in this process.

Backward compatibility

• Existing storages (PRTG, Nagios, SenHub, HTTP/Prometheus) are unchanged. Adding an otlp block is purely additive.
• Probe configurations (type: cpu, type: redfish, etc.) work identically. The OTel mapping comes from the bundled YAML transformer definitions.
• No new probe types are required to push to OTLP — every existing probe routes through the OTLP strategy when it's configured.

Troubleshooting

Agent starts but nothing reaches the collector

Check the agent log for:

strategy.otlp WRN OTLP metrics export failed   error=...

Common causes:

• Wrong endpoint or unreachable host. The agent uses lazy gRPC connect — the failure surfaces only on the first push attempt (after metrics.interval).
• TLS handshake failure. Set insecure_skip_verify: true temporarily to confirm certificate verification is the cause.
• Authentication rejected by gateway. Check the collector / vmagent / Grafana Cloud auth log; verify the headers: block.

Metrics appear with unexpected names in VictoriaMetrics

The default VictoriaMetrics OTLP-to-Prometheus translation strips the senhub_ prefix you might see in the Prometheus exposition. That's intentional — the OTLP path uses the canonical OTel names (system.cpu.utilization, not senhub_system_cpu_utilization). Add resource_to_telemetry_conversion: enabled: true to ensure resource attributes (service.name, deployment.environment) are preserved as labels.

Some metrics are missing

Check the agent log for:

strategy.otlp WRN Metric has no OTel mapping — not exported via OTLP.

Same policy as the Prometheus path: a probe metric without an otel: block in its YAML definition is silently skipped, and the agent warns once per (probe_type, metric_name). Add the mapping or otel.skip: true directive in definitions/<probe>.yaml and restart.

Logs aren't arriving

Verify:

1. signals.logs.enabled: true in the storage params.
2. The collector pipeline includes the otlp receiver in the logs: pipeline (not just metrics:).
3. The probes you expect to produce logs are configured: syslog, event, linux_logs, filetail, windows_eventlog, snmp_trap, or an otlp_receiver with signals: [logs]. Other probes do not produce log records.
4. The receiver tolerates the OTel log data model; older versions of vmagent or Loki collectors may need a recent build.

Reverse the agent → push direction

If you want a pull-based scrape model instead of push, use the Prometheus endpoint — same data, same names, different transport.

What the dependency discovery can and cannot see

The discovery reads the socket table on each emission scrape. It sees a connection that is open at the moment of a scrape. It does not see a flow that opens and closes entirely between two scrapes, however many times that happens: nothing on the host records it by then. An absent edge therefore means "not observed", never "does not exist", and that distinction matters before concluding from a map that two machines do not talk to each other. A shorter interval narrows the blind spot; it does not close it.