Skip to content
ISE

Kubernetes

Kubernetes has no alerting layer of its own, so ISE supplies one: the integration syncs cluster state into the estate and runs ISE’s own deterministic detectors over it, raising observation signals for conditions worth a human’s attention. Evidence queries and a tiered action catalogue support investigation and governed remediation on the same cluster.

Capabilities

Section titled “Capabilities”

Discovery. The cluster, its namespaces, workloads (Deployments, StatefulSets, DaemonSets), and nodes are enumerated into the estate, with Kubernetes labels flowing into ISE’s unified tag pool. Containment relationships (workload in namespace in cluster) come with discovery, and cross-source identity joins nodes onto hosts known from DataDog or the cloud integrations. Namespaces sync roughly every five minutes and nodes every ten by default.

Observations. Deterministic detectors run on a slow loop over synced state — never a model’s opinion, each with a fixed confidence reflecting how certain the rule is:

  • crashloop — a container in CrashLoopBackOff
  • oom_kill — a container killed for exceeding memory
  • pending_pod — a pod stuck unschedulable
  • unhealthy_workload — a workload short of its desired ready replicas
  • node_not_ready, node_pressure, node_flapping — node conditions worth attention

Observations flow onto incidents through the same severity-and-confidence rules as any other signal, and recover when the condition clears.

Evidence. On demand during an investigation: describe_pod, pod_logs, recent_events, pending_pods, rollout_status, node_capacity. Fetched when asked for, recorded on the incident.

Actions. The governed catalogue, each through the tiered approval pipeline:

Action Tier Notes
set_label T1 Writes a label at the source (fix-at-source tagging).
restart_rollout T1 Rolling restart of a workload; reversible by nature.
scale_workload T1 Scale within policy limits.
edit_resource T2 Modify a resource definition; requires approval.
delete_resource T3 Destructive; always requires a human approver.

Setup

Section titled “Setup”
  1. Prepare a kubeconfig for the cluster with a dedicated service account. A read-only deployment needs only get/list/watch on namespaces, pods, nodes, events, configmaps, deployments, statefulsets, daemonsets, and replicasets — no access to secrets.
  2. In ISE, add a Kubernetes integration (Settings → Integrations) and paste the kubeconfig YAML; optionally name a context if the file defines several. The credential is validated at store time (malformed YAML and unknown contexts are rejected immediately), encrypted at rest, and redacted from logs.
  3. The health check runs immediately; the System card shows connection status and last sync per slice.
  4. Grant write RBAC to the service account only if you intend to enable actions.

One integration per cluster: entity identity is cluster-scoped, so several clusters appear side by side in the estate without colliding.

Examples

Section titled “Examples”

A crash loop becomes an incident. A bad image ships and a Deployment’s pods enter CrashLoopBackOff. The crashloop detector raises a high-confidence observation attributed to the workload entity; an incident opens, already carrying the namespace/cluster context and the workload’s labels.

Governed remediation. Investigating that incident, evidence shows the previous ReplicaSet was healthy. A restart_rollout (T1) — or an edit_resource (T2) reverting the image — is proposed on the workload, approved per its tier, executed by ISE, and the whole trail lands in the audit log. When the pods go healthy the observation recovers and the incident resolves.