Secret Store¶

The agent can resolve ${secret:NAME} references in its configuration from an OS-native, on-disk secret store, so that passwords, tokens and connection strings never sit in plaintext in agent.yaml, probes.d/*.yaml or strategies.d/*.yaml.

${secret:NAME} sits alongside the ${env:} and ${file:} substitutions documented in Configuration → Environment and File Substitution. It is resolved at config load time, before any probe or strategy starts.

The `${secret:}` reference¶

Syntax	Resolves to
`${secret:NAME}`	The value stored under `NAME` in the secret store. Aborts boot if the name is unknown or no backend is available.
`${secret:NAME:-fallback}`	The stored value, or `fallback` if the name is unknown.

A missing required ${secret:} reference (no default) aborts agent boot with the offending name in the error message — a referenced secret that has gone missing must never silently resolve to an empty string. The error carries only the secret NAME, never its value.

The store is only touched when a ${secret:} reference actually exists: a configuration that uses no ${secret:} never creates a key file or store on disk.

probes:
  - name: veeam-prod
    type: veeam
    params:
      endpoint: "https://veeam.example.com:9419"
      username: monitor
      password: "${secret:veeam-prod.password}"

Backends per OS¶

The active backend is selected automatically per platform. Each stores its data next to the agent configuration file.

Platform	Backend	Store files (in the config directory)
Linux (default)	`age-keyfile`	`agent-secret.key` (X25519 identity, `0600`), `secrets.age` (ciphertext)
Linux (systemd, hardened)	`systemd-creds`	`creds.d/*.cred` (one encrypted credential per secret)
Windows	`dpapi`	`entropy.bin` (per-install entropy), `secrets.dpapi` (ciphertext, sealed with Windows DPAPI machine-local scope)
macOS / other (dev)	`age-keyfile`	`agent-secret.key`, `secrets.age`

On Linux, the age-keyfile backend is the default because it works unprivileged from any context and needs no systemd wiring. The systemd-creds backend is the hardened opt-in and is selected when any of the following holds:

SENHUB_SECRET_BACKEND=systemd-creds is set (explicit admin opt-in, e.g. for the seal step outside the unit);
a populated creds.d/ store already exists;
$CREDENTIALS_DIRECTORY is set (the daemon runs under a unit that wired LoadCredentialEncrypted=) and no secrets.age store exists in the configuration directory. An existing age store is kept, so a unit credential added for another purpose does not orphan it.

SENHUB_SECRET_BACKEND wins over the other rules. It accepts systemd-creds or age; SENHUB_SECRET_BACKEND=age forces the age key-file store, even under a unit that sets $CREDENTIALS_DIRECTORY.

With systemd-creds, the service reads its secrets through the credentials systemd hands it at start, which the drop-in /etc/systemd/system/senhub-agent.service.d/10-senhub-credentials.conf declares, one LoadCredentialEncrypted= line per sealed secret. The agent keeps that drop-in in line with creds.d/ by itself:

agent install and agent refresh-unit write it when creds.d/ holds secrets and remove it when it holds none, then reload systemd;
agent secret migrate --wire-unit wires the secrets it has just sealed;
agent uninstall removes it with the configuration.

A host already on the age store keeps its secrets there when it moves to systemd-creds: while both stores exist the agent reads creds.d/ first and the age store for the names it does not hold, so the move can be made one secret at a time, each re-stored with sudo SENHUB_SECRET_BACKEND=systemd-creds /usr/local/bin/senhub-agent secret set <name>, before the age files are removed.

After a secret set or secret rm, run sudo /usr/local/bin/senhub-agent secret wire-unit (or sudo /usr/local/bin/senhub-agent refresh-unit) so the next start sees the change. None of these restart the service: run sudo /usr/local/bin/senhub-agent restart to load the new credentials.

The store files are owned by the account the agent runs as and readable only by it (age key file 0600, DPAPI files restricted to SYSTEM + Administrators). Because the store is root-/service-owned, resolving a sealed secret requires the same privilege as the agent itself — see the privilege note under agent key show.

The `agent secret` command¶

agent secret manages the store. A secret VALUE is never accepted on the command line (it would leak via ps or shell history); set reads it from a hidden prompt, from standard input, or from a file. The command reads and writes the root-/service-owned store, so it runs behind the agent's privilege gate.

Command	Description
`secret set <name>`	Store or replace a secret. The value comes from a hidden prompt (interactive), a line of stdin (piped), or `--from-file <path>`.
`secret get <name>`	Print a secret value to stdout — a deliberate reveal. Warns when the output is not a terminal.
`secret list`	List secret NAMES only (never values).
`secret rm <name>`	Delete a secret.
`secret migrate`	Move inline plaintext secrets out of the config into the store and rewrite them as `${secret:}` references (see Sealing inline secrets). With `--wire-unit`, then wire the systemd-creds drop-in.
`secret status`	Show the active backend and store location and the number of stored secrets.
`secret wire-unit`	(Linux/systemd-creds) Regenerate the unit credential drop-in from `creds.d/`, or remove it when the store is empty.

All subcommands honour --config-path <path> to locate a non-default configuration directory. On Linux, run them as root with the full path of the installed binary, as below; on Windows, from an elevated prompt.

# Store a secret from a hidden prompt
sudo /usr/local/bin/senhub-agent secret set veeam-prod.password
# Secret value (hidden): ********
# stored secret "veeam-prod.password" in age-keyfile; reference it as ${secret:veeam-prod.password}

# Store from a file (no value on the command line)
sudo /usr/local/bin/senhub-agent secret set db.password --from-file /root/db-pass.txt

# Store from stdin (piping)
printf '%s' "$DB_PASS" | sudo /usr/local/bin/senhub-agent secret set db.password

# List names and show backend status
sudo /usr/local/bin/senhub-agent secret list
sudo /usr/local/bin/senhub-agent secret status

`agent key show`¶

agent key show prints the agent key, the key PRTG, Nagios and a Prometheus scrape read the agent with. It does not open the web console, which answers only the administration key (senhub-agent console opens it). It loads the configuration and prints the resolved key, so it works whether the key is still inline or has been sealed into the store as ${secret:agent.key}.

Resolving a sealed key reads the root-/service-owned store, so key show runs behind the same privilege gate as agent secret — it is not a read-only command.

sudo /usr/local/bin/senhub-agent key show

Sealing inline secrets into the store¶

Sealing also happens automatically on first boot

From config_version 3, the agent seals inline plaintext secrets into the store automatically the first time it starts: it moves each value into the store, rewrites the field to a ${secret:...} reference, and stamps the file config_version: 3. The bump to version 3 happens only when a secret is actually sealed — a secret-free configuration stays at version 2 and is untouched. agent secret migrate is the explicit, on-demand equivalent of that boot-time step (for example to seal a configuration before distributing it). See Configuration → Config versions.

With the systemd-creds backend the service cannot do this step: it runs as a non-root account, and only root can encrypt with the host key. It then leaves the inline values in place and logs once, at start, Inline secrets left in place: the secret store seals only as root. Seal them with sudo /usr/local/bin/senhub-agent secret migrate --wire-unit and restart the service.

agent secret migrate scans the configuration for fields whose NAME denotes a secret (password, passphrase, secret, token, api_key, community, credential, dsn, uri, private_key) and whose value is still an inline plaintext (not already a ${...} reference). It moves each value into the store and rewrites the field to a ${secret:} reference. Identifier-style fields such as user, login and email are deliberately left alone.

Before:

probes:
  - name: veeam-prod
    type: veeam
    params:
      endpoint: "https://veeam.example.com:9419"
      username: monitor
      password: "S3cr3t!"

Run the migration:

sudo /usr/local/bin/senhub-agent secret migrate
# sealed inline secrets into the store and rewrote them to ${secret:} references

After — the plaintext is gone from the config and lives only in the encrypted store:

probes:
  - name: veeam-prod
    type: veeam
    params:
      endpoint: "https://veeam.example.com:9419"
      username: monitor
      password: "${secret:veeam-prod.password}"

The store key is derived from the owning instance name and the field path (for example veeam-prod.password, or citrix-01.director.auth.password for nested fields), so every sealed value has a stable, human-readable reference.

Secret Store¶

The ${secret:} reference¶

Backends per OS¶

The agent secret command¶

agent key show¶

Sealing inline secrets into the store¶

The `${secret:}` reference¶

The `agent secret` command¶

`agent key show`¶