How to name a metric so tools read it right

Nobody reads a metric name except machines. When one is wrong they do not fail. They answer, and the number is wrong.

rate() decides what to do by looking at a suffix. A dashboard divides by a thousand because of a word. A Zabbix item key is built from the name and then collects history under it for months. Nothing in that chain validates the name, so a name that lies produces a graph instead of an error.

Prometheus and OpenTelemetry ask for the same things here. We got them wrong in 0.6.0 and fixed them in 0.6.1, which is where the examples below come from.

Base units

Seconds and bytes. The linter carries the table, and it is short:

amperes bytes celsius grams joules kelvin meters metres seconds volts

// and the ones that are not, with what they must become
minutes → seconds    hours → seconds    days → seconds
weeks   → seconds    bits  → bytes      calories → joules
fahrenheit → celsius

// each one again behind every SI and IEC prefix, so milliseconds,
// microseconds, nanoseconds and kibibytes are caught too
pico nano micro milli centi deci deca hecto
kilo kibi mega mibi giga gibi tera tebi peta pebi

The unit is implicit in the name, so it ends up in every threshold and every panel division written against it. When it varies between exporters, the conversion moves into other people's queries. An alert ported from an exporter that works in seconds onto one that works in milliseconds is wrong by a factor of a thousand and looks fine.

A certificate expiry in days is the case that survives review, because it reads well. The machine does not read it.

_total declares a type

Prometheus reserves _total for counters, the monotonic series you take a rate() of. A gauge carrying that suffix is read as a counter by everything downstream.

A gauge counting jobs, named ..._total, reads 12, then 9, then 14.

12  →  9  →  14

# a counter cannot go down, so the drop to 9 is read as a
# reset, and the 12 counted before it is added back in

reported increase:  (14 - 12) + 12  =  14
actual change:                        +2 jobs

Our own audit turned up something we had not expected. .total was standing for three different things at once, so the work was not renaming, it was deciding what each one meant:

None of them looked wrong next to the code that produced it.

The unit once, spelled out

Abbreviations are rejected on sight: s, ms, us, ns, sec, kb, mb. And a generated name that already carries its unit gets the canonical one appended on top of it.

senhub_redis_cmd_usec_microseconds_total
  an abbreviation, a non-base unit, and a counter suffix on a gauge

senhub_redis_ops_per_sec_per_second
  the source name already said per_sec; the generator added the rest

Where the rules collide

OpenTelemetry names a gauge that counts objects <object>.count. Prometheus reserves _count for histograms and summaries. We keep the OpenTelemetry spelling and exempt it, rather than invent a third name that neither ecosystem uses.

A linter works on words, so it will be wrong about some of yours. In senhub_netscaler_ns_throughput_bytes_per_second, ns is NetScaler's own word for the appliance, not nanoseconds. The exemption is written where the rule is applied:

// promlintExemptAbbreviation lists names where promtool reads a
// token as an abbreviated unit but the token is a vendor word:
// `ns` is NetScaler's own name for the appliance-wide scope,
// not nanoseconds.
func promlintExemptAbbreviation(name, token string) bool {
    return token == "ns" && strings.HasPrefix(name, "senhub_netscaler_ns_")
}

Checking it

Prometheus ships the linter and it reads an exposition on standard input.

$ curl -s localhost:9100/metrics | promtool check metrics

Run it. If your exporter is more than a year old it will find something.

An audit fixes the names once. Nobody chose the drift in the first place, so it comes back. We turned the lint into a test that walks every probe definition and the agent's own metrics, derives the Prometheus name each one will be exposed under, and applies the rules to it:

--- FAIL: TestExposedNamesPassPromlint
    veeam/jobs: senhub_veeam_jobs_total: a gauge ends in _total,
      which is reserved for counters
    redis/cmd_usec: senhub_redis_cmd_usec_microseconds_total:
      unit microseconds is not the base unit seconds
    redis/ops: senhub_redis_ops_per_sec_per_second:
      the rate unit appears twice

A second test feeds that guard one name of each broken class and fails if they get through. A naming guard is a handful of string comparisons, and it will go on passing quietly the day one of them stops matching.

If you have to rename

We did not export both names for a transition period. It doubles the series count for everyone, and nobody migrates while the old name still works.

An alert left on an old name does not go red. The series it reads no longer exists, so nothing is evaluated and nothing is signalled. The check is arithmetic: compare the number of series a rule reads before and after.

A fleet runs both versions during a rollout, so the bridging query needs its aggregation.

min by (httpcheck_target) (
  senhub_httpcheck_tls_expiry_seconds / 86400
  or senhub_httpcheck_tls_expiry_days
) < 14