Implementation of incident clients for various notification systems.
| Client | Module | Java Module |
|---|---|---|
| PagerDuty V2 Events API | incident-pagerduty |
software.sava.incident_pagerduty |
| incident.io Incidents V2 | incident-io |
software.sava.incident_io |
| incident.io HTTP Alert Sources | incident-io |
software.sava.incident_io |
| Generic Webhook / Slack / Telegram | incident-webhook |
software.sava.incident_webhook |
All depend on incident-core (software.sava.incident_core), which provides the
provider-neutral IncidentClient
API described below.
final var client = PagerDutyEventClient.clientBuilder()
.defaultClientName("CLIENT_NAME")
.defaultRoutingKey("INTEGRATION_KEY")
.createClient();
final var payload = PagerDutyEventPayload.build()
.summary("ex-summary")
.source("ex-source")
.severity(PagerDutySeverity.critical)
.timestamp(ZonedDateTime.now(UTC))
.component("ex-component")
.group("ex-group")
.eventClass("ex-class")
.customDetails("ex-num-metric", 1)
.customDetails("ex-boolean", true)
.customDetails("ex-string", "val")
.link(PagerDutyLinkRef.build()
.href("https://github.com/sava-software/incident-client")
.text("Sava Incident PagerDuty Event Client")
.create())
.image(PagerDutyImageRef.build()
.src("https://www.pagerduty.com/wp-content/uploads/2016/05/pagerduty-logo-green.png")
.href("https://www.pagerduty.com/")
.alt("pagerduty")
.create())
.create();
final var triggerResponseFuture = client.triggerDefaultRouteEvent(payload);
final var changeEventPayload = PagerDutyChangeEventPayload.build(payload).create();
final var changeEventResponseFuture = client.defaultRouteChangeEvent(changeEventPayload);
final var triggerResponse = triggerResponseFuture.join();
final var ackResponse = client.acknowledgeEvent(triggerResponse.dedupKey()).join();
final var resolveResponse = client.resolveEvent(triggerResponse.dedupKey()).join();
final var changeEventResponse = changeEventResponseFuture.join();PagerDuty limits Events API payloads to 512 KB, so the client refuses any request body over
PagerDutyEventClient.MAX_REQUEST_BODY_BYTES (524,288 UTF-8 bytes) with an
IllegalArgumentException before sending it.
The Events API authenticates with the routing key alone, so the client sends no Authorization
header; a custom endpoint that needs one sets it with extendRequest.
idempotencyKey and visibility are required by the API; a request without them is
rejected. Severities, incident types, statuses, roles, and custom fields are all
workspace-specific ids — look them up in your incident.io workspace.
try (final var httpClient = HttpClient.newHttpClient()) {
final var client = IncidentIoClient.clientBuilder()
.bearerToken("BEARER_TOKEN")
.httpClient(httpClient)
.createClient();
final var request = CreateIncidentRequest.requestBuilder()
.idempotencyKey(UUID.randomUUID().toString())
.name("Test Incident")
.summary("Test Java client")
.visibility(CreateIncidentRequest.Visibility.PRIVATE)
.mode(CreateIncidentRequest.Mode.test)
.severityId("SEVERITY_ID")
.incidentTypeId("INCIDENT_TYPE_ID")
.incidentRoleAssignments(List.of(new CreateIncidentRequest.IncidentRoleAssignment(
"INCIDENT_ROLE_ID",
CreateIncidentRequest.UserReference.byEmail("responder@example.com"))))
.customFieldValues(Map.of("CUSTOM_FIELD_ID", "value"))
.build();
final var response = client.createIncident(request).join();
System.out.println(response.reference());
System.out.println(response.permalink());
}customFieldValues is sugar for text custom fields; select, catalog, link, and numeric
fields use customFieldEntries with a matching CustomFieldValue — see the
module README. slackChannelNameOverride,
incidentTimestampValues, and retrospectiveIncidentOptions round out
IncidentsCreatePayloadV2.
An HTTP alert source deduplicates on a key and resolves with the same key; the workspace's alert routes decide whether the alert becomes an incident. The client authenticates with the alert source's own secret, not an API key.
final var client = IncidentIoAlertEventClient.clientBuilder()
.alertSourceConfigId("ALERT_SOURCE_CONFIG_ID")
.token("ALERT_SOURCE_SECRET")
.createClient();
client.sendAlertEvent(AlertEvent.build()
.title("Disk full on db-1")
.status(AlertEvent.Status.firing)
.deduplicationKey("db-1-disk")
.metadata("service", "db")
.create()).join();
client.sendAlertEvent(AlertEvent.build()
.title("Disk space recovered on db-1")
.status(AlertEvent.Status.resolved)
.deduplicationKey("db-1-disk")
.create()).join();See the module README for the metadata value model and the provider-neutral adapter.
incident-webhook POSTs a structured message to a single configured webhook endpoint —
fire-and-forget notification, not incident management. WebhookFormats.GENERIC_JSON
(provider id webhook) sends a canonical JSON document of the alert for receivers that
do their own mapping; WebhookFormats.SLACK_TEXT (provider id slack) sends a plain
{"text":"..."} Slack incoming-webhook message; TelegramTextFormat (provider id
telegram) sends a Bot API sendMessage body to
https://api.telegram.org/bot<TOKEN>/sendMessage with a configured chatId and,
optionally, a forum topic (messageThreadId) and severities sent silently
(silentSeverities); its link previews are always disabled, and an alert to a basic group
that was upgraded to a supergroup is sent once more to the supergroup, with a WARNING
logged (see the module README).
final var client = WebhookClient.clientBuilder()
.endpoint("https://hooks.slack.com/services/T000/B000/XXXX")
.createClient();
final IncidentClient incidentClient = client.incidentClient(WebhookFormats.SLACK_TEXT);Service-level code can be written against IncidentClient and switch providers via
configuration. IncidentClients creates a client from configuration alone — provider
modules on the module or class path register themselves via ServiceLoader, and the
provider config value selects one:
incident.provider=pagerduty
incident.routingKey=INTEGRATION_KEYincident.provider=incident.io
incident.bearerToken=BEARER_TOKEN
incident.visibility=private
incident.severityIds.CRITICAL=SEVERITY_IDincident.provider=incident.io-alerts
incident.token=ALERT_SOURCE_SECRET
incident.alertSourceConfigId=ALERT_SOURCE_CONFIG_ID
incident.resolveTitle=Recoveredfinal IncidentClient incidentClient = IncidentClients.createClient(properties, "incident");The JSON equivalent wraps the provider's config object, with provider first:
{"provider": "pagerduty", "config": {"routingKey": "INTEGRATION_KEY"}}Adapters can also be created in code from a native client:
final IncidentClient incidentClient = usePagerDuty
? pagerDutyEventClient.asIncidentClient()
: incidentIoClient.incidentClientBuilder()
.visibility(CreateIncidentRequest.Visibility.PRIVATE)
.severityId(IncidentSeverity.CRITICAL, "SEVERITY_ID")
.createClient();
final var alert = IncidentAlert.build()
.summary("Validator missed its leader slot")
.details("No block produced for slot 350000000.")
.severity(IncidentSeverity.CRITICAL)
.source("validator-01.example.com")
.timestamp(ZonedDateTime.now(UTC))
.customDetail("slot", 350000000L)
.create();
final var response = incidentClient.reportIncident(alert).join();
if (incidentClient.supportsResolve()) {
incidentClient.resolveIncident(response.key()).join();
}A correlation key longer than the client's declared limit (IncidentClient#maxKeyLength():
255 characters for PagerDuty, none for incident.io and the webhook formats) reaches the
provider shortened by IncidentClient.boundKey, to the key's first characters, a colon and
the SHA-256 of the whole key, the same way on every report and resolve, so an alert may be
keyed by whatever identifies its subject and resolved by that key or by response.key().
IncidentService.createService(incidentClient) wraps a client with retries driven by a
retry-delay function. When a provider asks for a wait before the next attempt, such as a
429's Retry-After (in seconds, or as a date measured from the response's own Date
header), the service waits it out before retrying, up to a limit of one minute
by default (IncidentService.DEFAULT_MAX_RETRY_AFTER); a longer wait is not served: the call
ends with the provider's failure after the failed attempt's own retry delay, whatever the
retry count, so an alert is never held for an
hour in silence and the caller can fall back. IncidentService.createService(incidentClient, maxRetryAfter) sets another limit, and Duration.ZERO refuses every such wait. A client
that throws where it should return a failed future, as the PagerDuty adapter does for a
request body over its size limit, fails the future the service returns, so a refusal by
the client arrives the way a failed request does.
The incident.io adapter maps IncidentSeverity values onto workspace severity ids
supplied at build time, and has no programmatic resolve: supportsResolve() returns
false and resolveIncident(String) fails the returned future with an
UnsupportedOperationException. It also needs an incidentTimestampId before it can
send IncidentAlert#timestamp() — incident.io timestamps are workspace ids — and carries
source() and customDetails() as text appended to the incident summary, since
incident.io custom fields are workspace schema objects rather than an arbitrary
key/value map. Provider-specific features — PagerDuty links,
images, and change events; incident.io custom fields and role assignments — remain on the
provider clients.
The incident.io alert-source adapter (incident.io-alerts) sends a firing event keyed by
the alert's key, with severity, source, timestamp and custom details in its metadata,
and resolves by sending a resolved event with the same key. That resolve addresses the
alert's lifecycle: whether an incident is created from the alert, and whether it closes
when the alert resolves, depends on the workspace's alert routes, so supportsResolve()
does not promise that an incident closes. Every event needs a title, and nothing
upstream says whether a resolve event's title replaces the alert's, so a resolve sends the
configured resolveTitle, or the key when none is set.