Keycloak Monitoring via Zabbix
LiveWhy Zabbix, not another Prometheus stack
Keycloak exposes its telemetry as a Prometheus/OpenMetrics endpoint, and the
reflexive answer is a Prometheus + Alertmanager deployment. But Zabbix was
already running infrastructure monitoring, and its native
Prometheus check preprocessing can parse an exposition-format
payload directly. So the whole pipeline is one importable template: identity
monitoring lands in the same dashboards and the same on-call rotation as
everything else, and nothing new gets installed on the Keycloak host — the
master item is an HTTP agent check against
{$KEYCLOAK.METRICS.URL}, not a local agent.
One scrape, many items
A single master item (keycloak.metrics.raw, type
HTTP_AGENT) fetches the full metrics output every minute with
history: 0 and trends: 0 — the raw text payload is
never stored, only handed to dependent items. Each dependent item applies a
PROMETHEUS_PATTERN preprocessing step with a label filter and
sum aggregation — for example
keycloak_response_errors{code=~"4..",route=~".*token.*"} isolates
failed token requests from machine clients — followed by
CHANGE_PER_SECOND to turn cumulative counters into rates. One
request against Keycloak per minute, regardless of how many series are
extracted. The example items use the aerogear-SPI metric names
(keycloak_failed_login_attempts,
keycloak_response_errors); the README documents swapping the
patterns for the http_server_requests_seconds_* series that
modern Quarkus builds expose with KC_METRICS_ENABLED=true.
What triggers fire
Every rate trigger uses min(...,5m), so the rate has to stay
elevated across the whole five-minute window — one noisy scrape can't page
anyone. The failed-login threshold lives in a template macro,
{$KEYCLOAK.FAILED_LOGINS.WARN} (default 30 per five minutes),
and the expression divides it by 300 to compare against the per-second item —
tune the number without touching the trigger. Service-to-service auth errors
above 0.1/s and 5xx rates above 0.05/s both fire at HIGH: a quietly rotated
client secret breaks machine auth without a single user complaint. And
nodata(...,5m) on the master item catches the metrics endpoint
going dark, because the monitoring is itself a dependency.
The Zabbix items deliberately sum across realms — coarse rates for paging.
Entity-level questions (which IP, which client, which realm) are
handled by two Sigma rules against Keycloak's event log in the SIEM: a
failed-login burst rule keyed on LOGIN_ERROR /
invalid_user_credentials grouped by source IP, and a
CLIENT_LOGIN_ERROR rule whose write-up distinguishes one failing
client (rotated secret) from many (client-ID enumeration). Both map to
ATT&CK T1110.
Delivery and CI
The Google Chat media type is a ~45-line Zabbix webhook script: it receives
{ALERT.SUBJECT}, {EVENT.SEVERITY},
{EVENT.STATUS}, and {HOST.NAME} as parameters, maps
severity to a plain-text prefix ([ALERT],
[WARNING], [RESOLVED]), and posts a
{"text": ...} body via HttpRequest. A non-200
response throws, so delivery failures surface in Zabbix's media log instead
of disappearing. CI is a yamllint job
(ibiqlik/action-yamllint@v3) on every push and pull request —
default ruleset with line length raised to 200 and Zabbix-style
yes/no/on/off allowed as truthy values — a syntax gate for the
template and Sigma rules before Zabbix's own import validation.