Use kx in CI

The kx diag and kx scan commands both sweep a namespace and print a table. In a pipeline nothing reads the table, and both would exit 0 whatever they found. Two flags change that. (kx tree and kx top take --json too, for the ownership graph and the usage listing; only the two that produce findings take --fail-on.)

kx diag -A --fail-on critical    # 0 if the cluster is healthy, 2 if not
kx scan -n prod --fail-on high --json

--json

The same analysis as a machine-readable document — every resource swept, healthy ones included, and every CVE behind the severity counts.

kx diag --json | jq '.resources[] | select(.verdict == "critical") | .name'
kx scan --json | jq '.images[] | select(.counts.critical > 0)'

The kx diag --json document has one shape whether the command was pointed at an index or a namespace, so that first expression reads kx diag 1 --json too — an indexed run is a sweep of one, and counts itself that way.

It is built from the same values the terminal and --html render, so the three views cannot disagree about what is wrong. A sweep serializes every resource regardless of --full: that flag governs how much of a table fits on a screen, and nothing is scrolling past a machine.

The document carries a schemaVersion, because this is a public surface the moment it ships — something will parse it in a pipeline, and a field moving underneath that is worse than one it can check for.

{
  "schemaVersion": 1,
  "namespace": "prod",
  "window": "24h",
  "checked": 12,
  "healthy": 1,
  "resources": [ … ]
}

The window field is how far back that run was allowed to look, spelled the way --since reads it. A run with no window carries no such field. It is what tells two runs of the same job apart when the second one is quieter — the cluster got better, or the window got narrower — which a document that only counted what it found could not say.

All four commands that emit --json name their subject with the same fields, so a pipeline that reads one does not have to learn a second shape to read the others. A sweep carries namespace, or allNamespaces: true for -A; an indexed run carries kind, name and namespace:

{
  "schemaVersion": 1,
  "kind": "Deployment",
  "name": "api",
  "namespace": "prod",
  "images": [ … ]
}

A resource that was named by index carries that index, so a consumer finding something worth acting on can name it — kx diag 4 — without listing again. A resource named by a mark carries mark instead, because a mark has no position in a listing; the two never appear together, and a document built from neither carries neither.

The kx tree --json document names every node with kind and name rather than the rs/web-7d8f label the terminal draws, carries the same index the tree printed, and always returns a roots list — one entry for an indexed resource or a single namespace, one per namespace for -A. A pod’s containers appear as children with a name and no kind, because a container is part of a pod rather than a resource of its own.

The kx top --json document reports percentages as numbers, and as null where there is none — a pod with no limit set has no percentage, and 0 would read as idle. Its resource field says whether the listing was pods or nodes, since a pod’s percentage is against its limits and a node’s against its capacity.

Severities are lower case throughout — critical, high, medium, low for image findings, critical, warning, healthy for verdicts — which is exactly what --fail-on accepts, so a value read out of a document can be typed straight back at the gate.

--fail-on

Turns either command into a gate.

The threshold is inclusive: --fail-on high fails on high and critical. It is validated before the cluster is read, so a typo fails immediately rather than after a sweep has already run.

An image whose scan failed breaches every threshold, for the same reason a missing test is not a passing one: an image kx could not read has not been shown to be clean.

Old news doesn’t hold the gate red

Evidence drives a finding, a finding drives the verdict, and the verdict drives the gate — so by default one FailedScheduling from three weeks ago, or one OOMKill a container recovered from last month, fails the job forever on a cluster with nothing currently wrong with it. The --since flag bounds what the report is allowed to look at, and only ever hides what finished — what is still going wrong is always reported, however long it has been going wrong:

kx diag -A --fail-on warning --since 24h   # today's failures only
kx diag -A --fail-on warning --since 7d    # a week's worth
kx diag -A --fail-on warning               # everything, however old

A gate is the place this matters most: without --since, a job that went red once stays red until somebody deletes the evidence.

A schedule longer than the window wants --since widened: a weekly CronJob whose last run failed six days ago needs --since 7d to fail the gate on it.

Set it once with diag_max_age in config.toml, or KX_DIAG_MAX_AGE in the job’s environment. Events are the least of it — most clusters discard those after an hour anyway; what a window really reaches is a container’s own history, which the kubelet keeps for as long as the pod lives.

The exit code is 2

Two, not one, and the difference is the point:

A pipeline that treated any non-zero as “unhealthy” could not tell a sick cluster from an unreachable one.

Publishing a report and failing on it

The --fail-on flag is independent of how the findings are presented. It applies alongside --json and --html alike, so a job can publish a report and still fail on what is in it:

kx diag -A --fail-on critical --out diag.html

The --out flag writes the page and returns, so the gate runs and the file is there for an artifact step to pick up — and it implies --html on its own, so there is no need to pass both. Plain --html without --out serves the report and blocks until Ctrl-C, which is right at a terminal and wrong in a pipeline — nothing sends Ctrl-C to a CI job, so it would hang until the runner killed it.

A whole cluster

kx diag -A --fail-on critical
kx scan -A --fail-on high

With -A, kx scan resolves every unique image in the cluster and scans each once, two at a time. The bound is memory rather than cores — a scanner unpacks an image and walks every package in it — so a wide sweep is steady rather than fast, and does not thrash a small runner.

Waiting before the next step

The kx wait command blocks until a resource is ready, which is the step between applying a change and checking it:

kx get jobs -m migrate
kx wait 1 --timeout=5m      # exits 0 on Complete, 1 the moment it fails
kx diag --fail-on critical

Without --for, the condition comes from the kind: a Pod or Node Ready, a PersistentVolumeClaim Bound, a Job Complete, a LoadBalancer Service given an address. A failed Job ends the wait at once with its reason, where kubectl wait --for=condition=Complete would run to the timeout. For a Deployment, StatefulSet or DaemonSet, kx rollout status is the wait.