Signals & incidents

Everything ISE reacts to is one of two things:

Signals are transient and machine-owned. They reflect current system state and come and go on their own. Nobody resolves a signal; it is present or it isn’t.
Incidents are durable and human-owned. An incident is the unit of work: once opened, a person manages it through to resolution, and it does not vanish because the signals underneath it did.

That split exists precisely to absorb the transience of signals. Monitors flap; the incident holds steady and carries the history.

Alerts and observations

Signals come in two kinds, and the distinguishing axis is who owns the definition of “bad”:

An alert is a pre-declared failure state detected by the source’s own monitoring — a DataDog monitor, a CloudWatch alarm, an Azure Monitor rule. ISE forwards it; it does not re-derive it.
An observation is a condition ISE itself judged noteworthy, where the source has no detection layer of its own. Kubernetes has no “monitor” object, so ISE runs its own detectors there.

The governing rule: ISE defers detection to a source when that source has one, and detects for itself only where the source does not. Because an observation is ISE’s opinion rather than a declared rule, it carries a confidence as well as a severity.

Severity and confidence

One canonical ladder, and every signal maps onto it:

info < low < medium < high < critical

Each connector maps its native levels onto that ladder (DataDog Alert → high, Warn → medium, and so on). A source’s own scale is usually a subset — DataDog cannot natively produce critical — which is exactly why normalising onto a superset ladder is worth doing.

Observations carry two axes, deliberately kept apart: severity is how bad, confidence is how sure. An ISE judgement can be severe but uncertain in a way an externally-declared alert never is.

When an incident opens automatically

Alerts are high-confidence by construction, so an alert auto-opens an incident when its effective severity meets the global threshold (for example ≥ high).
Observations need both severity at or above the threshold and confidence at or above the confidence bar. Below the bar they stay observations for a human to promote.

The threshold governs automation only. You can always open an incident by hand from any signal, however quiet.

Tuning noise

Four controls, at different altitudes — worth knowing which one you want:

Control	Where it acts	Effect
Ignore rules	On the integration, at ingest	A whole class of alert (matched on tags) never becomes a signal at all.
Downgrade	On a signal’s severity	Keep seeing it, keep it as evidence, stop opening an incident for each one.
Ignore	On a signal	Durable suppression: still recorded, never opens or reactivates an incident.
Silence	On a signal	Keeps its real firing status visible, suppresses promotion, and resolves the live incident.

Downgrading is a severity override: scoped (per integration, signal type, or entity), audited (the record says “downgraded high→medium by person”), and it never mutates the connector’s default — a connector upgrade cannot wipe your local tuning. It is also one-click from wherever the noise appeared, because tuning that requires a settings expedition doesn’t happen.

The incident lifecycle

New → Active → Resolved → (Reactivated | Closed).

Resolved means believed-fixed and can reactivate if the problem recurs within a configurable window. Closed happens automatically after an incident has been resolved for some days without recurring — and a recurrence after closing is a new incident, not a resurrection.

Correlation at ingest is deterministic and cheap — no AI. One signal maps to at most one open incident by its stable key, so a firing signal either opens an incident or attaches to the existing one, and a recovered-then-retriggered signal reattaches rather than spawning a duplicate.

Escalate, never de-escalate

A live incident escalates when its signal’s effective severity rises above it. It never de-escalates: a signal improving does not lower an incident’s severity or resolve it.

The asymmetry is deliberate. Escalation is information — the estate is worse than the record says, and saying so late is worse than saying so loudly. De-escalation would be a judgement, and standing an incident down is a human’s call. It also keeps a flapping monitor from swinging an incident’s severity, and its position in your queue, on every sync. A severity that only ratchets upward records the worst the estate got, not where it happened to be when you looked.

Acknowledgement is implicit

There is no “acknowledge” button, because doing almost anything substantive to an incident acknowledges it as a side effect: posting a message, triggering analyse/diagnose/propose, approving or executing a remediation, or setting the assignee. “Assign to me” is the explicit seen, mine gesture. Signal-scoped tuning — downgrade, ignore, silence — does not acknowledge, because those are verdicts about noise rather than about the incident.

Acknowledged is not resolved: it means someone picked it up.

Resolution cascades

Resolution flows from the incident to its signals, never the other way. The normal path is that signals go Recovered on their own, a human resolves the incident, and the signals then read Resolved. You can also resolve an incident while signals are still firing — and if such a signal is reported again it re-asserts, reactivating the incident.

One root cause often trips many monitors. Deciding that several incidents share one cause is a judgement, not an ingest-time fact, so ISE proposes a merge and a human accepts or rejects it — never an automatic merge. Merged children are listed under a master incident, their lifecycles frozen while merged, and a child’s escalation lands on its master.

The core loop — what happens to an incident once it exists.
Actions & approvals — how a fix gets applied.
Per-integration alert semantics live on each integration page.