0.5.4 — 2026-08-18¶

The daemon no longer runs as root and no longer holds a writable copy of itself, three topology identities are re-keyed once on upgrade, overlay networks become part of the topology, every entity type can finally be found in its own telemetry, a Docker Swarm probe arrives, Kubernetes coverage triples and gains cluster events, ntp measures the clock without a time daemon, container metrics no longer need host root, debug logging works for the first time, and the log file is written for a person to read.

Breaking changes¶

Local database identities are re-keyed once on upgrade¶

A database with no stable server-reported id falls back to an identity built from the address it answers on. On loopback that string is identical on every machine, so two MariaDB instances on two different hosts arrived in the topology as one entity: its version flipped between the two every few minutes, and its telemetry join keys resolved to one host while its attributes described the other.

The fallback is now scoped by the host:

Before	After
`127.0.0.1:3306`	`mysql:3306@<host.id>`
`127.0.0.1:6379`	`redis:6379@<host.id>`
`localhost:27017`	`mongodb:27017@<host.id>`

Databases reached over a routable address keep their identity unchanged — the address already distinguishes them, and re-keying them would cost a migration for no defect. The same is true of any instance you named yourself with instance_name, and of PostgreSQL, which reports a stable system identifier and never used the fallback.

What a topology consumer sees. The old entity is retired explicitly and an alias edge links the new identity to it, so the change reads as a decision rather than as an agent that went silent. Nothing is rewritten: the retired entity keeps its history, and a time-travel read at the moment of the switch shows both entities and the link between them. A current-state query will not show the alias — its target is retired by then — which is expected, not a failure.

Side effect worth knowing: until now a local database had no host. Its identity contained the loopback address, which the anti-collapse guard refuses to anchor, so the runs_on edge was never emitted. Scoped identities clear the guard honestly, and local databases now appear in their host's impact radius for the first time. (#740, #779)

Kubernetes count metrics no longer carry a `_ratio` suffix¶

Thirty-nine Kubernetes metrics declared the dimensionless unit 1. The OTel-to-Prometheus naming rule reads that as a ratio and appends _ratio, so a count of failed job pods reached Prometheus as k8s_job_failed_ratio and a replica count as k8s_statefulset_ready_ratio. Both are plain integers.

Before	After
`k8s_job_failed_ratio`	`k8s_job_failed`
`k8s_job_active_ratio` / `_succeeded_ratio` / `_desired_completions_ratio`	`k8s_job_active` / `_succeeded` / `_desired_completions`
`k8s_statefulset_ready_ratio` / `_desired_ratio` / `_current_ratio` / `_updated_ratio`	`k8s_statefulset_ready` / `_desired` / `_current` / `_updated`
`k8s_daemonset_ready_ratio` / `_desired_scheduled_ratio` / `_current_scheduled_ratio` / `_misscheduled_ratio`	`k8s_daemonset_*` without the suffix
`k8s_hpa_current_replicas_ratio` / `_desired_` / `_min_` / `_max_`	`k8s_hpa_*_replicas`
`k8s_node_ready_ratio`, `k8s_pod_ready_ratio`, `k8s_container_ready_ratio`	same names without `_ratio`
`k8s_resourcequota_hard_ratio` / `_used_ratio`	`k8s_resourcequota_hard` / `_used`

Counts now declare a unit naming what is counted ({pod}, {node}, {job}, {cpu}, {resource}) and one-hot state series declare {state}; annotation units carry no Prometheus suffix.

senhub_kubernetes_up_ratio was left alone at first, on the grounds that every probe's up declared unit 1 and breaking that alignment for one probe would trade a naming defect for a naming inconsistency across forty others. That reasoning was right, and it pointed at the real fix rather than at leaving it — so the whole set moved together instead. See Boolean and enum metrics lose their _ratio suffix below.

Dashboards and alerts querying the affected series need the suffix removed.

The agent is on disk once, and the daemon cannot write it¶

A hardened Linux install carried the agent binary twice: the copy you ran, and a senhub-owned copy under /var/lib/senhub-agent/bin that the unit execed. The second existed so the unprivileged daemon could rename a new binary over itself during auto-update.

It worked, and the shape was wrong — not because of the duplication, but because of what the duplication was for. The daemon is the part of the agent exposed to input you do not control: OTLP over the network, SNMP traps, syslog, tailed files, the responses of every target it probes. It is therefore the component most likely to be compromised, and it owned the executable systemd runs. Whoever compromised it got persistence across restarts.

The signature check did not close that. It ran inside the same process, so an attacker controlling the daemon controlled the code doing the checking. Verification performed by the party that may be compromised is not verification.

What changes. One binary, root-owned, at /usr/local/bin/senhub-agent (/usr/bin for future distro packages), inside ProtectSystem=full's read-only tree. The daemon no longer installs anything on Linux. It still checks for new versions when auto_update.enabled is set — it reports what it finds and names the command that applies it:

A newer version is available. The agent does not install it itself on Linux:
the binary is root-owned so the service account cannot rewrite it.
Apply it with 'sudo senhub-agent update', ...

Updating is sudo senhub-agent update <version> today, and apt/dnf/zypper once the packages are published. This is what package-managed agents do, and the package manager verifies against the system keyring and records what it installed.

Upgrading a host. sudo senhub-agent refresh-unit does it in one step: promotes the binary to the system path if nothing is there yet, repoints ExecStart, removes the old directory. It does the binary first on purpose — if that failed after the unit had been rewritten the service would come back as 203/EXEC, so a failure aborts with the old, working unit still in place.

Two things to know:

The unit and the binary move together. The hardened unit marks /var/lib/senhub-agent non-executable, so a host that receives the new unit while its ExecStart still points at the old copy fails to start. install and refresh-unit rewrite both at once; do not hand-copy one without the other.
refresh-unit normally preserves a custom ExecStart so a path you chose survives a refresh. The old /var/lib/senhub-agent/bin/senhub-agent is the single exception — it is recognised by name and repointed even though the file is still there, because it is not a path anyone chose. A genuinely custom path such as /opt/senhub/bin/senhub-agent is still left alone.

Windows is unaffected and keeps in-process updates: an MSI install stages a signed MSI and hands the upgrade to msiexec, a privileged installer outside the agent — the same separation, reached by a different road. A ZIP install replaces its own binary, which escalates nothing there because the service runs as LocalSystem. (#794)

Boolean and enum metrics lose their `_ratio` suffix¶

The OTel-to-Prometheus rule appends _ratio for the dimensionless unit 1, which is correct for a utilisation gauge in [0,1]. Every availability and state gauge in the agent declared that unit, so the endpoint exposed senhub_cassandra_up_ratio, senhub_db_up_ratio, senhub_windows_service_state_ratio and a hundred others. They are booleans and enumerations, not fractions.

The mapper already had the instinct — it skips _ratio for counters, with a comment saying enumerated booleans would mislead — but gauges fell through, and that is where all of these live.

118 Prometheus series are renamed. Percentages are unaffected: they declare the source unit %, the mapper divides by 100, and they really are fractions. Verified across all 250 declarations — not one describes a percentage without carrying %. Genuinely dimensionless quantities keep _ratio too: cache hit rates, compression and data-reduction ratios, memory fragmentation, AP satisfaction.

Nothing changes for PRTG, Nagios, the web UI or the SenHub cloud — they read channels, not an OTel unit.

The three consumers shipped with the agent move in the same release. The one that mattered most is packs/alerts/vmalert/, which carried senhub_icmp_up_ratio == 0 and friends: a rule matching a name that no longer exists stops firing in silence, which is the worst outcome available for an alert rule. The Grafana dashboards and the Prometheus metrics reference follow.

All 118 renamed series

| Before | After | |---|---| | `senhub_activemq_up_ratio` | `senhub_activemq_up` | | `senhub_ad_hybrid_sync_health_ratio` | `senhub_ad_hybrid_sync_health` | | `senhub_ad_hybrid_up_ratio` | `senhub_ad_hybrid_up` | | `senhub_apache_up_ratio` | `senhub_apache_up` | | `senhub_cassandra_up_ratio` | `senhub_cassandra_up` | | `senhub_ceph_health_status_ratio` | `senhub_ceph_health_status` | | `senhub_ceph_up_ratio` | `senhub_ceph_up` | | `senhub_chrony_up_ratio` | `senhub_chrony_up` | | `senhub_citrix_license_grace_active_ratio` | `senhub_citrix_license_grace_active` | | `senhub_clickhouse_up_ratio` | `senhub_clickhouse_up` | | `senhub_consul_leader_ratio` | `senhub_consul_leader` | | `senhub_consul_up_ratio` | `senhub_consul_up` | | `senhub_couchdb_up_ratio` | `senhub_couchdb_up` | | `senhub_db_mysql_replica_io_thread_running_ratio` | `senhub_db_mysql_replica_io_thread_running` | | `senhub_db_mysql_replica_sql_thread_running_ratio` | `senhub_db_mysql_replica_sql_thread_running` | | `senhub_db_postgresql_replica_io_running_ratio` | `senhub_db_postgresql_replica_io_running` | | `senhub_db_replication_health_ratio` | `senhub_db_replication_health` | | `senhub_db_replication_role_ratio` | `senhub_db_replication_role` | | `senhub_db_up_ratio` | `senhub_db_up` | | `senhub_db_version_info_ratio` | `senhub_db_version_info` | | `senhub_dns_up_ratio` | `senhub_dns_up` | | `senhub_docker_up_ratio` | `senhub_docker_up` | | `senhub_elasticsearch_cluster_health_ratio` | `senhub_elasticsearch_cluster_health` | | `senhub_elasticsearch_up_ratio` | `senhub_elasticsearch_up` | | `senhub_envoy_up_ratio` | `senhub_envoy_up` | | `senhub_exchange_online_service_health_ratio` | `senhub_exchange_online_service_health` | | `senhub_exchange_online_up_ratio` | `senhub_exchange_online_up` | | `senhub_exec_skipped_ratio` | `senhub_exec_skipped` | | `senhub_exec_timeout_ratio` | `senhub_exec_timeout` | | `senhub_haproxy_up_ratio` | `senhub_haproxy_up` | | `senhub_hardware_logical_disk_encrypted_ratio` | `senhub_hardware_logical_disk_encrypted` | | `senhub_hardware_physical_disk_has_active_operations_ratio` | `senhub_hardware_physical_disk_has_active_operations` | | `senhub_hardware_physical_disk_location_indicator_active_ratio` | `senhub_hardware_physical_disk_location_indicator_active` | | `senhub_httpcheck_content_match_ratio` | `senhub_httpcheck_content_match` | | `senhub_httpcheck_tls_valid_ratio` | `senhub_httpcheck_tls_valid` | | `senhub_httpcheck_up_ratio` | `senhub_httpcheck_up` | | `senhub_hyperv_ha_cluster_group_state_ratio` | `senhub_hyperv_ha_cluster_group_state` | | `senhub_hyperv_ha_cluster_node_state_ratio` | `senhub_hyperv_ha_cluster_node_state` | | `senhub_hyperv_ha_replica_health_ratio` | `senhub_hyperv_ha_replica_health` | | `senhub_hyperv_ha_replica_state_ratio` | `senhub_hyperv_ha_replica_state` | | `senhub_hyperv_ha_up_ratio` | `senhub_hyperv_ha_up` | | `senhub_hyperv_up_ratio` | `senhub_hyperv_up` | | `senhub_hyperv_vm_state_ratio` | `senhub_hyperv_vm_state` | | `senhub_ibmi_job_priority_ratio` | `senhub_ibmi_job_priority` | | `senhub_ibmi_jobs_topn_cap_hit_ratio` | `senhub_ibmi_jobs_topn_cap_hit` | | `senhub_ibmi_journal_active_ratio` | `senhub_ibmi_journal_active` | | `senhub_ibmi_library_list_position_ratio` | `senhub_ibmi_library_list_position` | | `senhub_ibmi_license_usage_limit_ratio` | `senhub_ibmi_license_usage_limit` | | `senhub_ibmi_netstat_interface_up_ratio` | `senhub_ibmi_netstat_interface_up` | | `senhub_ibmi_netstat_listener_up_ratio` | `senhub_ibmi_netstat_listener_up` | | `senhub_ibmi_ptf_group_installed_ratio` | `senhub_ibmi_ptf_group_installed` | | `senhub_ibmi_ptf_group_level_ratio` | `senhub_ibmi_ptf_group_level` | | `senhub_ibmi_sysval_audit_level_ratio` | `senhub_ibmi_sysval_audit_level` | | `senhub_ibmi_sysval_security_level_ratio` | `senhub_ibmi_sysval_security_level` | | `senhub_ibmi_watch_session_active_ratio` | `senhub_ibmi_watch_session_active` | | `senhub_icmp_up_ratio` | `senhub_icmp_up` | | `senhub_influxdb_up_ratio` | `senhub_influxdb_up` | | `senhub_ipmi_up_ratio` | `senhub_ipmi_up` | | `senhub_jenkins_up_ratio` | `senhub_jenkins_up` | | `senhub_kafka_up_ratio` | `senhub_kafka_up` | | `senhub_kubernetes_up_ratio` | `senhub_kubernetes_up` | | `senhub_memcached_up_ratio` | `senhub_memcached_up` | | `senhub_modbus_register_value_ratio` | `senhub_modbus_register_value` | | `senhub_modbus_up_ratio` | `senhub_modbus_up` | | `senhub_mongodb_up_ratio` | `senhub_mongodb_up` | | `senhub_mssql_ha_replica_connected_ratio` | `senhub_mssql_ha_replica_connected` | | `senhub_mssql_ha_replica_health_ratio` | `senhub_mssql_ha_replica_health` | | `senhub_mssql_ha_replica_role_ratio` | `senhub_mssql_ha_replica_role` | | `senhub_mssql_ha_up_ratio` | `senhub_mssql_ha_up` | | `senhub_nats_up_ratio` | `senhub_nats_up` | | `senhub_nginx_up_ratio` | `senhub_nginx_up` | | `senhub_ntp_leap_status_ratio` | `senhub_ntp_leap_status` | | `senhub_ntp_up_ratio` | `senhub_ntp_up` | | `senhub_nvidia_up_ratio` | `senhub_nvidia_up` | | `senhub_opensearch_cluster_health_ratio` | `senhub_opensearch_cluster_health` | | `senhub_opensearch_up_ratio` | `senhub_opensearch_up` | | `senhub_oracle_enterprise_up_ratio` | `senhub_oracle_enterprise_up` | | `senhub_os_updates_reboot_required_ratio` | `senhub_os_updates_reboot_required` | | `senhub_os_updates_up_ratio` | `senhub_os_updates_up` | | `senhub_phpfpm_up_ratio` | `senhub_phpfpm_up` | | `senhub_powerstore_appliance_state_ratio` | `senhub_powerstore_appliance_state` | | `senhub_powerstore_cluster_state_ratio` | `senhub_powerstore_cluster_state` | | `senhub_powerstore_drive_state_ratio` | `senhub_powerstore_drive_state` | | `senhub_powerstore_replication_state_ratio` | `senhub_powerstore_replication_state` | | `senhub_powerstore_up_ratio` | `senhub_powerstore_up` | | `senhub_powerstore_volume_state_ratio` | `senhub_powerstore_volume_state` | | `senhub_promscrape_up_ratio` | `senhub_promscrape_up` | | `senhub_proxmox_node_status_ratio` | `senhub_proxmox_node_status` | | `senhub_proxmox_up_ratio` | `senhub_proxmox_up` | | `senhub_proxmox_vm_status_ratio` | `senhub_proxmox_vm_status` | | `senhub_pulsar_up_ratio` | `senhub_pulsar_up` | | `senhub_rabbitmq_node_running_ratio` | `senhub_rabbitmq_node_running` | | `senhub_rabbitmq_up_ratio` | `senhub_rabbitmq_up` | | `senhub_redis_aof_enabled_ratio` | `senhub_redis_aof_enabled` | | `senhub_redis_cluster_state_ratio` | `senhub_redis_cluster_state` | | `senhub_redis_replication_role_ratio` | `senhub_redis_replication_role` | | `senhub_smart_disk_health_ratio` | `senhub_smart_disk_health` | | `senhub_snmp_up_ratio` | `senhub_snmp_up` | | `senhub_solr_up_ratio` | `senhub_solr_up` | | `senhub_sqlserver_database_status_ratio` | `senhub_sqlserver_database_status` | | `senhub_swarm_up_ratio` | `senhub_swarm_up` | | `senhub_systemd_unit_active_state_ratio` | `senhub_systemd_unit_active_state` | | `senhub_systemd_unit_load_state_ratio` | `senhub_systemd_unit_load_state` | | `senhub_systemd_unit_sub_state_ratio` | `senhub_systemd_unit_sub_state` | | `senhub_tcpdial_up_ratio` | `senhub_tcpdial_up` | | `senhub_tomcat_up_ratio` | `senhub_tomcat_up` | | `senhub_unifi_up_ratio` | `senhub_unifi_up` | | `senhub_varnish_up_ratio` | `senhub_varnish_up` | | `senhub_veeam_object_last_run_failed_ratio` | `senhub_veeam_object_last_run_failed` | | `senhub_vsphere_ha_nsx_edge_cluster_health_ratio` | `senhub_vsphere_ha_nsx_edge_cluster_health` | | `senhub_vsphere_ha_nsx_manager_health_ratio` | `senhub_vsphere_ha_nsx_manager_health` | | `senhub_vsphere_ha_up_ratio` | `senhub_vsphere_ha_up` | | `senhub_vsphere_ha_vsan_health_ratio` | `senhub_vsphere_ha_vsan_health` | | `senhub_wildfly_up_ratio` | `senhub_wildfly_up` | | `senhub_windows_service_state_ratio` | `senhub_windows_service_state` | | `senhub_winservices_up_ratio` | `senhub_winservices_up` | | `senhub_zookeeper_server_state_ratio` | `senhub_zookeeper_server_state` | | `senhub_zookeeper_up_ratio` | `senhub_zookeeper_up` |

(#791)

Two more topology identities are re-keyed once on upgrade¶

Same family as the database re-key above, and the same one-time effect: the old node is retired explicitly, a new one appears, and an alias edge links them so both timelines stay joinable.

winservices and chrony pinned a constant as their identity — winservices://localhost and chrony://localhost. Neither contains any host component, so the value was byte-identical on every machine: every host running the probe collapsed onto one node. Worse than a merge, because each host still drew its own edge to itself, so that single node fanned out to the whole fleet and joined the hosts transitively through it. The identity is now winservices@<host.id>, following the contract for a local thing with no stable id of its own.

Monitored processes were keyed on {process.pid, process.creation.time} — unique on one machine, colliding the moment two hosts start a process with the same pid at the same second, which is what a fleet booted from one image does. The identity gains host.id.

Neither has any entity in our own production graph, so for us these are mines defused rather than migrations. A fleet that runs winservices on Windows estates will see the re-key.

Without a host id, neither probe now emits anything at all: the identity is built from it, and inventing one is what produced the collapse. A visible gap beats a node that is wrong on every machine.

Changed¶

The log file is written for a person to read¶

The agent's log file was JSON, and worse than dense: the secret-masking layer re-encodes each entry through a map, and Go sorts map keys alphabetically — so the timestamp landed at the end of every line and the level sat somewhere in the middle. Nothing aligned from one line to the next.

{"count":116,"level":"info","message":"Successfully sent datapoints",
 "module":"data_store","strategy":"http","time":"2026-08-13T11:12:58+02:00"}

is now

2026-08-13 11:12:58.102 INF Successfully sent datapoints module=data_store count=116 strategy=http

Date first, then level, then the sentence, then the structured fields. Console output carries the date too — a line pasted into a ticket without its date cannot be correlated with anything. Timestamps gained real sub-second precision; the previous format stored whole seconds and rendered a constant .000.

Secret masking is unaffected: the formatter sits between the masker and the file, so the masker still redacts structured fields rather than pattern-matching formatted text.

Set --log-format json (or SENHUB_LOG_FORMAT=json) to restore the machine-parseable form for shipping the file to an aggregator. The remote log shipper is unaffected and stays JSON. (#772)

Telemetry ingested by `otlp_receiver` keeps the sending application's identity¶

An application pushing to the agent's OTLP receiver sends its own resource — its service.name, its host.name. Until now the agent replaced that identity with its own on the way out, so several applications behind one agent arrived at the backend indistinguishable: a log sent with service.name=my-app was stored under the agent's service.name, and an ingested metric carried two different values for service.name in a single export — the agent's on the resource, the application's on the datapoint — leaving the backend to silently keep one.

The three signals now honour one contract on the OTLP output: the emitting application's resource is forwarded as sent, and the agent's context (tenant, site, environment, and the telemetry.relay.* identity that names which agent relayed the data) is only ever added to keys the sender left unset. Traces already worked this way; logs and metrics now match them.

Nothing changes for PRTG, Nagios, Prometheus, the web UI or the SenHub cloud. Those sinks read tags, not an OTLP resource, and ingested telemetry still reaches them exactly as before — the sending application's resource attributes keep arriving as tags.

If a dashboard or query relies on ingested logs carrying the agent's service.name, point it at the agent's own logs, or at telemetry.relay.instance.id, which names the relaying agent without overwriting the sender. (#765, #767)

New¶

`ntp` measures the clock against a reference, with no time daemon involved¶

A new free ntp probe reports how wrong the local clock is, by exchanging NTP packets directly with reference servers you name. It reads no local time daemon and does not need one.

Until now the only answer was the chrony probe, which covers hosts running chrony and reports what the daemon believes about the clock it steers. A host on systemd-timesyncd, ntpd, the Windows Time service or nothing at all reported nothing — and a daemon synchronised to a wrong source reports an offset near zero with complete confidence.

- name: ntp
  type: ntp
  params:
    servers:
      - ntp1.example.internal
      - ntp2.example.internal

Works on Linux, Windows and macOS, needs no software on the host and no privileges. It emits ntp.time.offset alongside ntp.round_trip.delay — the delay is the confidence attached to the offset beside it, because the measurement assumes both directions of the exchange took equally long.

senhub.ntp.state names why a measurement is missing: unreachable, refused, unsynchronised or invalid_response. The common real failure is not a drifting clock but a firewall closing on outbound UDP 123, which arrives as unreachable — so alert on the state, not only on the offset.

There is no default server and the probe is not enabled by default. A default would point every agent at somebody else's infrastructure, and it would be the wrong measurement anyway: the useful comparison is against the reference the host is meant to follow. The default interval is five minutes and samples are capped, for the same reason.

The probe page documents what the measurement supports and what it does not. It reliably answers "is this clock wrong enough to break something", at the scale of tens of milliseconds and up; it is not a precision instrument. (#788)

Container metrics without granting host root¶

The docker probe needs /var/run/docker.sock, which is root:docker 0660, so on a hardened non-root install it collected nothing. The usual remedy — join the docker group — is root on the host by another route, since anyone who reaches the socket can start a container that mounts /.

The kernel exposes most of it without any of that. When the socket is unavailable the probe now reads container counters straight from cgroups: 26 of its 31 metrics on a default hardened install, with no group, no capability and no path to root. CPU, memory, block I/O and process counts all come from world-readable files.

What the fallback cannot provide, and does not invent:

per-container network counters, which live in the container's network namespace rather than its cgroup — reading them means entering that namespace or asking the daemon;
the restart count, which is daemon bookkeeping;
names, images, labels and state, so containers are identified by id alone.

senhub.docker.source says which source produced each cycle, socket or cgroup, because the two do not carry the same fields and an operator comparing container bandwidth across hosts needs to know which of them cannot report it at all.

cgroup v1 splits controllers across sibling trees and needs a different reader; there the fallback declines rather than publishing half the picture, and says so. (#797)

Overlay networks are topology, not just labels¶

A Docker Swarm overlay is the reachability boundary: two workloads on the same segment address each other by name, two on different segments cannot, whatever the firewall says. Until now it rode as metric labels — queryable, not traversable — so "can A reach B" meant reading several docker network inspect outputs on the right node.

Overlays are now emitted as network.segment entities, declared by the cluster that owns them and joined by the workloads on them:

service.instance --has_segment--> network.segment    (the manager declares)
container        --attached_to--> network.segment    (the node observes)

The manager sees which segments exist; the nodes see who is on them, so a complete picture needs the swarm and docker probes both running. A manager-only deployment reports segments with no attachments, which is a stable state and not a defect.

Only swarm-scoped overlays produce an attachment. A bridge network is local to one engine and identically named on every host, so a segment built from it would be a node shared by the whole fleet.

Membership is a necessary and not sufficient condition for reachability: network policies restrict on top of it, and the probe still measures attachment, never traffic.

Every entity type can now be found in its own telemetry¶

An entity you cannot pivot to telemetry is a picture, not a tool. Measured on a real graph at the start of this cycle, that pivot worked for three entity types out of ten.

It now works for all of them. Each type either stamps the identity it is keyed on onto the datapoints that describe it — db.instance.id on database metrics, k8s.pod.uid alongside the pod name, the VM GUID as vmid, the process pid with its creation instant, network.segment.id on segment metrics — or is explicitly declared as having no telemetry of its own, so a consumer stops looking instead of guessing.

Identity labels are omitted while unresolved rather than emitted blank: an empty label and the real one are two series for one subject.

Containers and pods say what they are, not just what they are called¶

A host carries its full nameplate. A container carried four attributes and a pod three — findable by name, described by almost nothing.

Containers now carry what their orchestrator states about them: the Swarm service and task, the Compose project and service, and their creation time. Pods carry the workload that owns them, read from the owner reference rather than parsed out of the name, with a ReplicaSet reported as the Deployment above it — nobody thinks in ReplicaSets.

Container labels are read through a whitelist, never copied wholesale: they are arbitrary operator input, and passing them through would let anyone inflate the graph with unbounded keys.

Docker Swarm cluster probe¶

A swarm probe reports cluster state from a manager node: nodes and quorum, service convergence, task lifecycle, and the overlay segments that decide what can reach what. Free tier.

It answers what no single host can. Quorum — a swarm that has lost manager quorum keeps every container running while silently refusing every change: no deploy, no rescheduling, no scaling, and from inside one node nothing looks wrong. Convergence — running replicas are counted from tasks, because Swarm publishes no running count and a service can declare five replicas while five tasks sit in rejected. Overlays — one series per (service, segment) pair carrying the service's virtual IP, so "can A reach B" is a lookup rather than three docker network inspect calls on the right node.

Traffic volume between services is deliberately not reported: the Engine API exposes no per-peer counters, and per-container interface counters cannot be attributed to a named overlay. The probe maps who can reach whom, not how much flows. (#757)

Kubernetes coverage raised to the k8sclusterreceiver standard¶

The kubernetes probe goes from 19 to 55 metric definitions, and gains two things it never had.

Cluster events on the log rail. Kubernetes events are the only place the cluster explains why a metric moved — "Failed to pull image", "0/5 nodes are available: 5 Insufficient cpu", "Back-off restarting failed container". They arrive as logs carrying the same join labels as the metrics, so a reader pivots from "why did this pod restart" to that pod's series without parsing the sentence.

Pods and containers in the topology, with the chain container → pod → host, so a node going down takes its pods, which take their containers, and an impact query answers transitively.

Storage, quotas and autoscaling are covered too: a claim stuck Pending, a namespace at 99 % of quota and an autoscaler pinned at max are each the reason a deployment will not scale, and none of them is visible from a workload's own replica counts. (#756)

A probe can be turned off without deleting it¶

enabled: false stops a probe and keeps its configuration:

probes:
  - name: mysql-prod
    type: mysql
    enabled: false
    params:
      host: 127.0.0.1

Deleting the entry worked before and still does, but it takes the credentials, intervals and tags with it. Omitting the key means enabled, so existing configurations are unaffected. A disabled probe collects nothing and reports no topology — it never appears as a monitored-but-broken target — and flipping the flag on a config reload stops it without restarting the agent. agent config check lists disabled probes explicitly. (#774)

The OTLP pipeline reports what it relays¶

The relays had no success counter: an operator could watch senhub_agent_otlp_receiver_ingested_total climb with no way to tell telemetry the agent had forwarded from telemetry it had accepted and never sent. A field report spent half a day on that ambiguity — the agent was relaying correctly the whole time.

Three counters close it, on /info/otlp, in senhub-agent status --otlp and on the Prometheus endpoint:

senhub_agent_otlp_spans_relayed_total
senhub_agent_otlp_logs_relayed_total
senhub_agent_otlp_metrics_relayed_total

Each counts only what the collector accepted, never a refused export, and pairs with the receiver's ingest counter for the same signal: equal totals mean everything ingested left the agent.

The HTTP receiver's startup log also names every route it serves. It previously logged only the metrics path even when the logs and traces handlers were mounted, which read as "traces is not wired" at exactly the moment an operator checks that. (#764)

`snmp_poll` collects IPv6 routes¶

Route collection walked only ipCidrRouteTable (RFC 2096), which is IPv4-only, so an IPv6 or dual-stack router surfaced none of its IPv6 routing table.

snmp_poll now also walks inetCidrRouteTable (RFC 4292), the address-family-agnostic successor, and emits IPv6 destinations as network.route entities exactly like IPv4 ones — canonical CIDR identity, host bits zeroed, RFC 5952 form (2001:db8:abcd::/48, ::/0 for the default route).

Devices implementing both tables list their IPv4 routes twice; the first entity per destination wins and the IPv4 table is walked first, so IPv4 route identities are unchanged. A device implementing only one of the two tables is normal and no longer treated as a failure — collection fails only when neither table answers, and the error then names both causes.

Zoned address families (ipv4z, ipv6z) are skipped: their destination is only meaningful inside one scope and would collide with its unzoned twin. (#716)

Fixed¶

Outbound dependency discovery needs root, and now says so¶

entities.depends_on_enabled derives "service A depends on B:5432" by mapping each outbound socket to the process that owns it. On Linux that mapping reads /proc/<pid>/fd, which only the owner may read — so the non-root daemon this release makes standard sees every other service's connections with no owner at all, and can emit nothing for them.

Measured on a host with the agent running as senhub:

Established sockets	Attributed to a process
45, seen as root	45
45, seen as the service account	0

The rail did not fail — it reported that nothing depended on anything, and a topology consumer retires dependencies by absence, so upgrading with this option enabled would have removed every dependency edge the consumer held.

Three things changed. A scrape that can attribute no socket but the agent's own is now reported as a failed observation rather than an empty one, so the consumer keeps its last good view instead of being told the dependencies ended; what eventually expires is marked unmonitored, not terminated. The agent logs the situation once at startup, naming the cause and the remedy. And the debounce became symmetric: an edge that takes three scrapes to appear now survives three missed ones, where a single miss used to retract it — which is what made an occasional missed observation reach the graph as an edge flapping in and out.

If you run this option and want the whole host's dependencies, the agent has to run as root. Left as-is on a non-root install, the rail reports only the agent's own outbound dependencies, and says so in the log rather than silently. (#808)

`chrony` produced no measurement at all, on any host¶

Reported from the field on 0.5.3:

chronyc: parsing stratum: strconv.ParseFloat: parsing "109.190.177.205": invalid syntax

chronyc -c tracking emits fourteen comma-separated fields, with the reference address at index 1 and the stratum at index 2. Every index in the parser was one position too low, and the length check demanded thirteen fields instead of fourteen — so it read the reference address as the stratum and shifted every value after it.

The report noted the failure appeared only once a host was synchronised, which is accurate and was the clue: an unsynchronised chrony leaves the address column empty, so there is nothing to misread. The impact is wider. On an unsynchronised host the parser read that empty column as the stratum and failed just the same. No ntp.* series was ever emitted by this probe, in any state, on any host — senhub.chrony.up=0 was its only output.

The test that should have caught this used a hand-written thirteen-field line with no address column, a shape chronyc does not produce. It proved the parser matched the invention rather than the tool. The fixture is now a verbatim capture from chrony 4.5, the two reported cases are regression tests, and a thirteen-field line is rejected instead of misread. Parse errors name the field index and the offending value.

The report also observed that nothing distinguished "this host has no chrony" from "the probe cannot read the chrony it has" — one needs action, the other does not. senhub.chrony.state answers it, one series per reason with exactly one set to 1: ok, not_installed, exec_failed, parse_failed.

The probe page gains the parameters it always accepted but never documented (chronyc_path, interval), offset thresholds tied to what actually breaks at each one, and a troubleshooting table per state. (#787)

Debug logging never produced what it advertised¶

Two independent defects, either one enough to make the feature useless, both present since module logging was written.

--filter <module> produced nothing at all, ever. Selective mode pinned the global log level to Info and then marked the chosen modules Debug — but the global level is a hard floor, checked before the logger's own level, so the filter's own selections were vetoed by the line above them. An operator asking for one noisy subsystem got silence, which reads as "that code path logs nothing".

--verbose reached sixteen modules out of a hundred and fourteen. The level map was treated as an allowlist, so a module absent from it stayed mute even with debug enabled globally. That map was frozen years ago, which means every probe added since — kubernetes and swarm included — was invisible under full verbose.

Measured on a real agent, forty-second runs: no flag, 0 debug lines; --verbose, 1355 lines across 12 modules; --filter probe, 15 lines from probe.* only, without the 1238 strategy.http lines that drown the verbose output.

The runtime log-level endpoint is fixed as a side effect — raising one module to debug on a running agent hit the same floor. (#772)

Fixed Entities Kubernetes A cluster node and the agent running inside it produced two host entities instead of one. Kubernetes returns /etc/machine-id verbatim (32 hex characters) while the agent renders the same bytes as a hyphenated UUID — same machine, same file, two spellings, and a silent duplicate for every node of every cluster. (#762)
Fixed Entities Kubernetes A pod waiting to be scheduled has no node, so it carried no relation and was dropped before reaching the backend — silently removing from the topology exactly the pod an operator is looking for. It now anchors to the cluster until it is placed. (#761)
Fixed Entities Network Network interface metrics now carry the identity tag that joins them to their interface entity; the entity existed but nothing in the metrics pointed at it. (#748)
Fixed Entities Docker Container metrics carried a shortened container id while the container entity is keyed on the full one, so the two could not be joined. (#758)
Fixed OTLP Relation attributes were built and then dropped before the wire, so an identity-alias edge arrived without the belief attributes that make a consumer act on it — the cost of sending it and none of the effect. (#779)
Fixed Auto-update Agents whose configuration predates 0.5.0 requested a doubled /releases/releases/ path and got a 404, so auto-update was dead on them — and they could not fetch the fix, because fetching the fix is what was broken. The URL is normalized on load. (#747)

`update` reconciles a pre-0.5.4 host that still has two binaries¶

Before this release a hardened Linux install carried the agent twice: the CLI copy in PATH, and the copy the systemd unit execed (/var/lib/senhub-agent/bin/senhub-agent), owned by the unprivileged daemon so it could replace itself during auto-update. sudo senhub-agent update <version> only ever replaced the copy it ran from, so the service kept running the old release while the CLI reported the new one — and the closing "Restart the agent to use the new version" made it look like the upgrade had landed.

update now reconciles both copies and names each file it wrote, and senhub-agent --version reports the skew instead of hiding it:

Version: 0.5.3 (commit: a8e67f7)
Service binary: 0.5.4 (/var/lib/senhub-agent/bin/senhub-agent)
Note: the systemd service runs a different build than this CLI binary.
      'sudo senhub-agent update <version>' updates both copies.

A service copy running a newer release than the one being installed is reported and left untouched rather than downgraded.

This matters for the upgrade itself and then stops mattering: 0.5.4 removes the second copy entirely (see The agent is on disk once). Once a host has migrated there is nothing left to reconcile and both behaviours go quiet, exactly as they always have on a single-copy install — legacy root unit, Windows, MSI. (#723)

`refresh-unit` no longer disarms a `--user root` install¶

senhub-agent install --user root writes a unit that runs with full privileges — which is the entire reason to choose it, for probes that need raw ICMP sockets or a privileged port. Running senhub-agent refresh-unit on such a host rewrote it to the hardened template with User=root, and the hardened template drops every Linux capability. The service kept starting, so nothing looked wrong; the active checks that needed those capabilities simply stopped working.

A refresh on a root install now produces the same unit the install produced, capabilities included. Root identity is expressed the way the installer expresses it — by the absence of a User= directive, systemd's default being root — instead of an explicit User=root on a capability-dropping unit.

Non-root installs are unchanged: the senhub user still gets the hardened unit verbatim, and a custom service user still gets it re-templated. A refresh still never switches a root install to the senhub user. (#689)

`filetail` can read the system log files again on a hardened install¶

On Debian and Ubuntu, /var/log/syslog and /var/log/auth.log belong to syslog:adm with mode 0640, so a filetail probe pointed at them collected nothing under the non-root unit — silently, with no error naming the cause. The journal was never affected: linux_logs reads it through the systemd-journal group the unit already grants.

senhub-agent install, senhub-agent refresh-unit and the .deb / .rpm postinstall now join the service user to the adm group, which grants exactly those log files and nothing else. Running refresh-unit is how an existing install picks it up. Where the group does not exist, the join is skipped and the install still succeeds.

The membership is granted through the user database rather than the unit's SupplementaryGroups=, because a SupplementaryGroups= naming a group absent from the distribution fails the unit at startup with 216/GROUP.

The admin guide now documents the grant, and warns against the workaround it replaces: CAP_DAC_READ_SEARCH does make the logs readable, but it bypasses every file read permission check on the host — /etc/shadow, private keys and any customer data included. On Red Hat systems rsyslog writes /var/log/messages as root:root 0600, where adm does not help and linux_logs is the answer. (#732)

Hosts opted into betas now converge to the stable release¶

A host running auto_update.include_beta: true resolved latest to the newest beta and stayed there — it never moved to the stable release that superseded it. A recette host opted into betas silently stopped tracking production.

Each channel is published with an alias record first, carrying the resolved version ({"latest", "0.5.3"}, {"latest-beta", "0.5.3-beta"}), and the merge of the two channels de-duplicates by version keeping the first record — so for the newest release the alias record is usually the only one left. Version selection then discarded records named latest, which made the newest stable release invisible, while the beta alias, named latest-beta, escaped the same filter and won.

Selection now looks at the version a record carries, never at the name of the record. A beta genuinely ahead of the newest stable still wins, so opting into betas keeps delivering them.

Stable hosts were never affected: they resolve latest through a different path. (#730)

No more registry warning on every HTTP push for the log conduit probes¶

filetail, linux_logs, windows_eventlog and snmp_trap publish their records straight to the log rail; the only datapoints they hand to the pull cache are their own throughput and health counters. Those probe types were not declared in the cache's discriminant-tag registry, so every HTTP-strategy push logged:

Probe type not in DiscriminantTagsRegistry - using no discriminant tags
  metric_name=senhub.filetail.records_emitted probe_type=filetail

The four types are now declared with an empty discriminant set, which is the correct shape rather than a gap: these counters carry no per-instance tag, and the probe name is already part of every cache key — so two filetail probes have always produced two distinct series, and still do. Only the log noise changes. (#724)

Security¶

The systemd unit is hardened, and you can check it yourself¶

The hardened unit stopped at dropped capabilities and ProtectSystem. It now carries the standard systemd restriction set: kernel tunables, modules, logs, cgroups and the clock become read-only; namespaces, SUID creation, realtime, personality changes and writable-executable memory are refused; system calls are limited to @system-service on the native ABI.

Socket families are narrowed to the four the agent actually opens — notably excluding AF_PACKET, which is raw frame capture on every interface and the family a compromised monitoring agent would want most. NoExecPaths covers the agent's own directories, so the write access it legitimately needs stops being a place to drop a payload and run it.

Measured with systemd-analyze security on Ubuntu 26.04 / systemd 259, shipped unit with no site drop-ins:

	Exposure
0.5.3	5.9 MEDIUM
0.5.4	2.0 OK

Run it yourself: systemd-analyze security senhub-agent. The least-privilege guide documents the exposure that remains and why each item stays — sockets, the process tree, device access — because a page claiming an agent is hardened is worth less than the command that proves it, and a reader who runs that command and finds unmentioned items would be right to distrust the rest.

Three directives are deliberately absent, with the reason written in the unit: PrivateDevices would hide the devices smart, nvidia and ipmi read; ProtectProc/ProcSubset would hide the process tree the process probe reports on; an IPAddressDeny allow-list would have to be rebuilt every time a probe is added.

Verified live rather than asserted: every probe on the test host kept collecting under the new restrictions, snmp_trap still binds UDP/162 with CAP_NET_BIND_SERVICE, and icmp_check still completes raw-socket pings with CAP_NET_RAW.

The guide also gains a measured account of what a non-root daemon does not see — three things, and everything else works — and what raising that costs, with CAP_DAC_READ_SEARCH named for what it is: unrestricted read of the filesystem, granted to a process that parses untrusted network input. (#794)

Security Dependencies Built on Go 1.26.6, which clears seven vulnerabilities in the standard library, all with reachable call traces from this agent: quadratic complexity in net/url, an unbounded count of post-handshake TLS messages, ReadHeaderTimeout not applied on the unencrypted HTTP/2 check, and missing recursion guards in encoding/xml and encoding/asn1. govulncheck reports no known reachable vulnerabilities in this release.

Known follow-ups¶

The swarm and docker probes reach the Docker Engine over a Unix socket and have no named-pipe support, so neither works against Docker on Windows — the platform most customers run. (#801)
Default probe configuration still covers four host probes; everything else on a machine is collected by nobody until someone writes YAML. (#777)
A probe can be disabled but not started or stopped at runtime — that needs a restart or a config reload. (#775)
The smart probe reports a clean cycle while reading nothing on a non-root install: /dev/sda is root:disk 0660, so it cannot open the device, and no warning names it. The disk group is the remedy and is documented; the silent success is not yet fixed. (#795)
process.open_file_descriptors is absent, correctly rather than zeroed, for processes owned by other users — /proc/<pid>/fd is owner-only. The omission is never explained in the log. (#796)
The docker cgroup fallback covers cgroup v2 only; on a v1 host the probe still needs the socket. (#797)
Relay enrichment is configured under signals.traces.relay_enrichment, but now governs relayed logs and metrics too; disabling it on the traces signal silently disables it for all three. The setting will move to a relay-level block, with the current key kept as a deprecated alias. (#766)
A metric point is stamped with the moment it is exported, not the moment it was observed. A series in the store is re-published every push cycle until it is evicted, so for up to one staleness_ttl window — ten minutes by default — an old measurement is presented as a current one. Eviction bounds how long that lasts; it does not stop it. In the same area, eviction runs only on the metrics push path, so with signals.metrics.enabled: false and persistence on, the checkpoint accumulates series with no producer indefinitely. (#812)